Status of the SuperKEKB
Accelerator Control System

M. Iwasaki (KEK)for the SuperKEKB accelerator control groupIntroduction

The KEKB accelerator is upgraded to SuperKEKB

x40 higher luminosity- More precise, reliable, and robust
accelerator control system is required.
- Many accelerator components are upgraded.

Reliable and robust network system is required.
Device controllers (software/hardware) for theaccelerator components should be upgraded.
2

What we did in JFY2013

1. New network system construction

Wider band-width (100MbE  10GbE),
Redundant system, New network configurations,
Wireless LAN system installations, …

2. Device controller upgrade & implementation
Vacuum system, LLRF, Magnet power supply,
Safety system, QCS cryogenic monitoring system, …

3. Additional new data archiving system
4. Beam abort system R&D
5. Timing system R&D
6. Renovation of the computing room3

SuperKEKB Control System• EPICS is used as the main software to control the accelerator
2 layer model• OPI (Operation Interface) --- operation programs on central servers• IOC (I/O Controller) --- equipment controls on frontend computers• Scripting Languages are used for the operation programs
SAD Script/Tk Python/Tk Tcl/TkConsoles
Central Servers

Accelerator Control Network
Frontend Computers
IOC (I/O Controller)

Field Buses

AcceleratorComponents~10,000 components# signals to control  ~200,000

SuperKEKB Control Network System
Nikko

Star network topology

Tsukuba- Main network switch (core switch)is located at the SuperKEKB CTL room.
- All network switches (edge switches)in the 26 sub CTL rooms connect tothe main network switch.

Fuji

Edge Switch

Oho
Edge Switch

SuperKEKBCTL room

Edge Switch

Edge Switch

Core Switch
Sub-CTL rooms

5Upgrade of the network system
In JFY2013, the control network system is upgraded.
1. Wider band-width data transfer (100MbE  10GbE)
Replace 100MbE switches to 10GbE ones.

2. Redundant system- Install additional optical cables.
- All edge switches are connected to the core switch via both
10GbE and 1GbE lines, and form the active/standby configuration.

3. Direct connection btw SuperKEKB and BelleII- Install optical cables to make direct connections.
- Install 10GbE switches for higher speed data transfer.

4. New network configuration- Apply VLAN segmentation into the SuperKEKB control network.
- Change Network connections btw SuperKEKB control network
and KEK laboratory network to enhance network security.

5. Wireless LAN system installation

SuperKEKB tunnel, AR, sub-control rooms, power equip. buildings

10GbE Network Switch Installation

We install ~30 10GbE switches in Feb. 2014,
and change the network configurations.
(Redundant connections, VLAN segmentations)KEKB-Belle Network Connection
D1

KEKB
CTLroom

Network
Switch(100MbE)

Media
Convertor(10MbE)(For Timing)
D2

For QCS
Tsukuba B4

Tsukuba
B4 / Electronics HutFor Belle

D2

Single-mode opt. cableMulti-mode opt. cable
LAN cable

Net. SW(100MbE)

Med.Conv(10MbE)

Net. SW(100MbE)

Tsukuba B3

Med.Conv(10MbE)For BPM
Tsukuba B4

SuperKEKB-BelleII Network Connection
D1Super
KEKB
CTLroom(For RF signal)
D2

D2

Single-mode opt. cable
Phase stabilized optical cable

Tsukuba B1(1GbE SW)
Tsukuba B3(10GbE SW)
Tsukuba B4(1GbE SW)

Belle solenoid/QCS
Cryogenic control

Belle II
QCS
TsukubaElectronics Hut

Tsukuba B3(10GbE SW)
Tsukuba B4(10GbE SW)
TsukubaElectronics
Hut

Belle IIBPMFor Timing9SuperKEKB-BelleII Network Connection
D1Super
KEKBCTLroom(For RF signal)
D2

Tsukuba B1(1GbE SW)
Tsukuba B3(10GbE SW)
Tsukuba B4(1GbE SW)

Belle solenoid/QCS
Cryogenic control

Belle IIQCS
TsukubaElectronics Hut- SuperKEKB & BelleII are directly connected
Tsukuba B3
Belle
II- Wider bandwidth
datatransfer(10MbE10GbE)
D2(10GbE SW)
Tsukuba B4
BPM(10GbESW)
Optical cables are installed in Dec. 2013
Tsukuba
Networkswitches
areinstalledinFeb.
2014
Single-mode opt. cable
Electronics
For Timing
Phase stabilized optical cable
10HutVLAN segmentation for SuperKEKB- We use EPICS to control the accelerator.
- EPICS uses UDP broadcasts to communicate btw IOC & OPI.
Many UDP broadcast packets in the control network.
(~90% of the total network traffic)- More UDP broadcasts in the SuperKEKB network.
Accelerator components with Ethernet interface also
receive the UDP broadcasts, and cannot work properly.
We apply VLAN segmentation
to protect the accelerator components
SuperKEKB control network consists of
1. EPICS network (for EPICS IOC and OPI)2. Device networks (for accelerator components)

VLAN segmentation for SuperKEKB
In Feb. 2014, we apply VLAN segmentation
for the SuperKEKB control network.
IOCDeviceDeviceIOCNetworkSwitch

EPICS Network
Device Network 1
Device Network 2Device

NetworkSwitch

IOCDevice

IOCDevice
NetworkSwitch

OPI

NetworkSwitch

Core Switch

We have tested the new VLAN segmentation
 IOCs in the EPICS network can control the devices in the device network.

Network reconfiguration toconnect with the KEK network
Internet

FirewallPCPC
PC User PC’s

SAD
computers

SAD

KEK Laboratory Network

ConsoleConsole

Firewall

NAS

SuperKEKB
Accelerator Control Network
IOCIOCIOCControl

EPICS IOC’s Accelerator control
PC’sSADSAD computers
for acc. control

We changed the networkconfiguration in Aug. 2013.
 Enhance the security
KEKB
Many computersconnect to both KEK and
KEKB control networks

SuperKEKB
Computers in acc. control
network don’t directly
connect to the KEK networkWireless LAN system installation
In JFY2012 – 2013, we installed wireless LAN into
- Whole SuperKEKB tunnel- AR tunnel- Power equip. buildings
for construction and maintenance of the accelerator.
All access points (~70) as well as Linac’s
are controlled by 1 controller.

14

Wireless LAN system
into the whole 3km tunnel area
Tsukuba

power equipmentbuilding

We install- LCX (Leaky CoaXial cable) antennas for arc
125m x 4 LCX antennas / one arc section

Power supply forarc section APs
is from the PoE atpower equip. buildings
Nikko- Collinear antennas for straight sections

4 collinear antennas / one straight section

LCX antenna
Access Point for LCX
Collinear antenna
15

LCX and Collinear antennas
in the SuperKEKB tunnel- Select LCX and Collinear antennas with radiation hardness >1MGy.
- APs and PoEs in the tunnel are installed within lead boxes.
- Connector connections are wrapped with lead sheets
and fixed with polyetheretherketone (PEEK) cable ties.

Device Controller for SuperKEKB
In KEKB, we mainly used VME/VxWoks IOC (I/O Contoroller)
In SuperKEKB, we use
1) VME/VxWorks IOC
CPU is upgraded

J. Odagiri

2) PLC/Linux IOC- Yokogawa FAM3 series- Linux running on theCPU module(F3RP61)
Apply for Vacuum system,
Beam collimators, LLRF,
Safety system, Magnet PS,…

3) PC/Linux IOC

CPU Module

I/O Modules

F3RP6117

Status of IOC Upgrade & Implementation• Full Replacement in progress– Vacuum Control– LLRF Control– Interface to PPS

VME IOC + CAMAC → PLC IOCVME IOC + CAMAC → µTCA IOC + PLC IOCVME IOC→ PLC IOC

• CPU Upgrade of VME IOC– Magnet PS– BPM– BTin progress
start soonplanning• VME IOC implementation– Beam Abort Systemin progress• PLC IOC development & implementation
 Next page18

Status of F3RP61 (PLC CPU module)
Implementation
J. Odagiri

FY2013• Interface to the Personnel Protection System• LLRF (used with µTCA system)• Vacuum system• Current stability controller for the high current Magnet PS
Things to do in FY2014• Interlock system for the Magnet PS system• Tilting Support (SX Magnet) Control• BT• Dumping Ring
19

T. T. Nakamura, A. Akiyama

ARCNET and Magnet Power Supply
synchronous start signal

ARCNET Interface board（4 ch. / boards）
PSICM

CPU(Power Supply Interface Controller Module)

magnetpower
supplyIOC(VME)

max. 20 nodes
twisted pair cable
local control
room

power supply
room

ARCNET（1 pair）
synchronous start signal（1 pair）
20

T. T. Nakamura, A. Akiyama

Version up of the PSICM(Power Supply Interface Controller Module)– Interface/Controller card plugged in power supply• Microprocessor is embedded• ARCNET controller and driver• Timing signal input to start synchronous ramping– New version of PSICM• Support high speed communication (10/5/2.5Mbps)• Support high resolution Power Supply (24/20/18/16bits DAC)
• Fully compatible to the current version for the Magnet PS• Redundant timing signals• More reliable connectors

Old Version

New Version(Prototype)21

T. T. Nakamura, A. Akiyama

Magnet PS Control System• FY2013 Status– New PSICM• Development of the prototype ver.2• Final tuning of the firmware• First mass production (1000 pcs.)→ done→ in progress→ in progress• FY2014 Plan– New PSICM• Second mass production (2000 pcs.)→ depends on budget• Installation– IOC/OPI Software development• modification from KEKB version
22

Data Archiving System- KEKBLog as a primary data archiving system (file based logging system)- CSS(Control System Studio)-based Archiver + PostgreSQL
as the 2nd option data archiving system

CSS installed users PCs can be remotely access to the database
For real-time / historical / trend monitoring

Data Archiving System

From October 2013, we monitor the QCS cryogenic systemwith the new archiving system (CSS + PostgreSQL).
 Store ~300 points data every 5 seconds.

Beam Abort System

T. Naito, A. Akiyama, S. Sasaki

We have developed the faster response Beam Abort System for SuperKEKB
E/O conversion, optical cable to transfer the signal, remove low-pass filters
 Response time improved from 100µs to 20µs

Loss monitors etc.
20 Local(over 130 points) Control rooms

SuperKEKB
Control room

VME
Abort
moduleAbort input
VME
Abort Transfer
module Cables(<3km)......

VME
Abort
moduleAbortKicker25

Beam Abort System- JFY2013

T. Naito, A. Akiyama, S. Sasaki

System check using the prototype system.
Abort module- JFY2014

E/O converter

Installation of the new beam abort system.

Timing System for positron injection

H. Kaji, K.Furukawa (Details in the Kaji-san’s talk)

In SuperKEKB, we construct the positron Damping Ring
 Positron injection timing scheme become complicatedTo account for DR, new timing system for e+ injection is required

K. Furukawa
27

Wire Scanner System

H. Kaji

Wire scanners are used as beam profile monitors at Linac and BT.
Control System is upgraded
1. CAMAC retired.

2. Event Receiver is installed.
- It enables the multi-beam operations.
- The 1st IOC using the new version ofFPGA and firmware⇒ This experience is helpful for
upgrade of the Main Trigger Station.
3. New functions are integrated- Change wire speed during the scan.
- Capability for multi-mode operation- Synchronization with BPM on-going⇒ for collection of beam position andcurrent, pulse-by-pulse.
Three systems have already installed and two of them are operated.
Installations of a few more wire scanners are planned.

28

Renovation of the computing room
Before the renovation

29

Renovation of the computing room

Last summer, we removed old server racks,
old electrical panel boards, power and signal cables.

Renovation of the computing room

We removed old cables (3t in total).

Renovation of the computing room
Electrical panel boards are renewedRenovation of the computing room
New server racks in the computing roomSummary
Upgrade of the accelerator control system
for SuperKEKB is in progress
In JFY2013, we did- New network system construction- Device controller upgrade & implementation- Additional new data archiver construction- Beam abort system R&D- Timing system R&D- Renovation of the computing room34

SummaryIn JFY2014, we should do- Network system for damping ring
- New computing server system- Device controller upgrade & implementation- Beam abort system installation- Timing system upgrade- Renovation of the control room console system- Accelerator operation equipment
New alarm system, task launcher, operation panels, …

for the 1st SuperKEKB operation in 2015.
35

Backup slides

36

Wireless LAN in the SuperKEKB tunnel

We measured the wireless LAN network speed in the SuperKEKB tunnel.
LCX effective transmission rate (Mbps)

TsukubaResults in this area

OhoTsukuba

Oho

Good network speed performance of ~18Mbps
in the whole tunnel area37

Wireless LAN system
Installation into AR

Collinear Antenna

Access Point

38

Wireless LAN system
into power equip. build.

Access Point

39

PLC Embedded IOC

EPICS on PLC
VxWorks CPU was available on PLC (Yokogawa, Mitsubishi)
Besides normal sequence / ladder CPU

Yokogawa starts to provide Linux (2.6) on PLC CPU (F3RP61)
Brave enough to choose open source environmentWe negotiate with Yokogawa to remove any license issues

Odagiri/KEK, Uchiyama/SHI-RIKEN, Yamada/KEK made much effort torealize EPICS implementation, (but no need for asynchronous records)
Takuya-Nakamura/MSC-KEK, et al, tailored the environment for KEKB
Procserv, pcmon, NFS, …

Four F3RP61-based IOCs are used in KEKB operation
Three since September 2008, and another later, four in totalBeam mask controllers and Pulsed-quad controllers
No trouble at all, they run more than 8 months

~20 new IOCs are also used in J-PARC operation now

KSTAR - KEK EPICS Meeting

Kazuro Furukawa, KEK, Jun.2009.

41

PLC Embedded IOCF3RP61 (e-RT3 2.0)
Linux 2.6.24
PPC 533MHz
128Mbyte RAM
100BaseTx x2
USBIEEE1394
SerialKEKB Beam mask
PCIcontroller
I/O Bus for FAM3 Module Interface
can access to mature FAM3 I/O Modules
Can be combined with conventional ladder CPU
Software development environment (ELDK)KSTAR - KEK EPICS Meeting

Kazuro Furukawa, KEK, Jun.2009.

42

PLC Embedded IOC

Simple Usage under EPICS
Conventional PLC usage
OPICA
Clients

with asynchronous accessIOC(Logics)

PLC usage with F3RP61

LadderCPU(Logics)

FAM3 PLC
I/O Modules

with only synchronous access and maybe with sequencer

OPICAClients

F3RP61IOC

FAM3 PLC
I/O Modules

F3RP61IOC

Ladder FAM3 PLC
CPU I/O Modules

If necessary, we can combine
OPICAClients

KSTAR - KEK EPICS Meeting

Kazuro Furukawa, KEK, Jun.2009.

43

Number Of F3RP61-based IOC
Group

Expected numbers（at T = 0）

RF (including LLRF)

13 + 2

Vacuum14

Magnet/Magnet Power
Supplies

11 + 3

Personnel Protection
System8

Beam Monitor

1+(?)

BT

2+(？)

Dumping Ring?

Helium Refrigerator

1

Total> 39 + ( ? )