Control System of the SPring-8 Storage Ring
R. Tanaka, S. Fujiwara, T. Fukui, T. Masuda,
A. Taketani, A. Yamashita, T. Wada and W. Xu
SPring-8, Kamigori, Ako-gun, Hyogo 678-12, JapanThe SPring-8 storage ring is under construction and will be commissioned in February 1997. The design for the control
system of the storage ring adopted the so-called Standard Model concept. VMEbus is used as the front-end control
system and is widely distributed around the 1436m circumference of the storage ring. The VMEbus is controlled by asingle-board-computer with a RISC processor. The VME crate is connected to the FDDI backbone network through an
FDDI-Ethernet switching hub. The CPU boards use the LynxOS-based HP-RT real-time operating system. The remote
I/O system (RIO), interconnected by optical fibers, is used as the field bus. The operator console system consists ofUNIX workstations with a human-oriented interface built using a Motif-based GUI builder. The control software designis based on the event-driven client-server scheme. The remote procedure call (RPC) is used for communication between
machine control applications over the network. Accelerator operation history and equipment parameters are stored in astandard database, using an RDBMS.

1. Introduction
The SPring-8 project is to construct a large scale advanced synchrotron radiation facility and to promote fundamental
science in the field of synchrotron radiation research. The facility being constructed at Harima Science Garden City inwestern Japan consists of a 1 GeV linac, an 8 GeV booster synchrotron and an 8 GeV low emittance storage ring. The
storage ring has 38 beamlines from insertion devices and 23 from bending magnets, including eight beamlines 300m longand three of 1000m. Long beamlines are available for medical experiments planned for the future. Furthermore, SPring-8
is reserving the space of four long straight sections in the ring for future options. The construction of the storage ring is
proceeding towards commissioning, which is scheduled for February 1997 [1].

2. Storage ring control
2.1 Concept
The design concepts of the SPring-8 control system are as follows:(1) distributed processors are linked with each other by a high-speed network,(2) sub-system (linac, synchrotron, and storage ring) control computers are loosely coupled, due to the different
accelerator construction schedules and for the convenience of independent operation,(3) all the accelerators are operated from one control room by a small number of operators,(4) the system is built using industry standard hardware and software as much as possible,(5) for high productivity of application programs, computer-aided program development tools are used.
2.2 Architecture
We designed the SPring-8 control system based on the above concept. Figure 1 shows the control system architecture.
It consists of UNIX engineering workstations (EWS), VMEbus systems, and optical fiber network (FDDI) systems [2]. In
the storage-ring control system the network structure is designed to be flat. The remote I/O systems (RIO) are usedmainly as the field bus system [3]. We use HP 9000/700 series EWS with the HP-UX operating system as operator
consoles and HP J200 for the database servers. The CPU board of the VME system is HP9000/743rt with a RISC
processor. The real-time OS HP-RT was chosen for the CPU board. The control software is designed based on the eventdriven client-server scheme. Machine control programs are all written in the C language. The remote procedure call(ONC/RPC) is used for communication between machine control applications over the network. Accelerator operation
history and equipment parameters are stored, using Sybase as the RDBMS. A detailed description is given in thefollowing sections.

Control room

L,Sy control system

Operator consoles
Prog. dev.

X-Term.

Utility 1
Safety 1

EWS

x4

x7

EWS

VME X-Term. EWS

ServerSecondaryServerPrimary

L, Sy prog. dev.

L, Sy control

EWS

Ethernet

EWS
VME

6portsÅcVME
49 VME's

RouterEther Switch
FDDI Node System

FDDI Node System

SR A-Zone

SR B, C, D-Zone

FDDI Node System

FDDI Node System

Ether Switch

Ether Switch

Ethernet
Hub
Magnet Interlock
PLC Network
Magnet PS

VME VME VME

Hub

EthernetHub
VME VME

VME VME VME

Hub

VME VME

Mag1 RF3 Mon1
RIO
Qa, St-PS

Con6

B.L.1

Vac1

Mag2 RF4 Mon2
RIO
B, Q, S, St-PS

Con6

X3

B.L.1

RIO

Vac1
RIO

Fig. 1. The architecture of the control system.
3. Hardware and software
3.1 VME system
Expandability and maintainability are the most important criteria for adopting industry standard hardware.
Consequently the VMEbus system was chosen for the front-end control system, as shown in Fig. 2. The interspersed
VME system is suitable for the widely distributed control environment of a large storage ring. The CPU board
HP9000/743rt/64, powered by the PA-RISC 7100LC processor, provides quite satisfactory performance (77.7MIPS). It
has 16MB of memory and a 20MB PCMCIA flash ROM card, which is used as a boot device for the operating system.
VME I/O modules such as DI, DO, TTL DI/O, AI, and PTG are used for direct control of RF equipment. The remote I/O(RIO) system is widely used as the field bus for magnet power supplies, BPMs, and vacuum system controllers. The
GPIB bus is used as a field bus as well. We will use 28 VME systems for the storage ring control.
3.2 Front-end operating system
The selection criteria for the OS on the VME system are mainly real-time performance and open system features,indicating compliance with the POSIX standard. From among such OSs HP-RT was chosen. HP-RT is the LynxOS
migration to the PA-RISC and can be called a real-time UNIX, so it is easy to understand and manage. The maindifference between LynxOS and HP-RT is the program development environment, since HP-RT is supported only in a

cross-development environment. HP-RT is running on the particular CPU boards developed by HP, so althoughavailability maybe limited, we do expect excellent support from the company. This is a crucial point for reducing
development cost and time for the new OS. HP-RT device drivers for VME modules are under development by thecontrol group [4].
FDDI
FDDI Node
Switching Ethernet(optical fiber)
Switching Ethernet(optical fiber)

Switching Ethernet(optical fiber)

I/O BoarI/O BoarI/O Boar

HP9000/743

GP-IB bus

RS-485RIO Network(optical fiber)

GP-IB device
Slave Ca

Equipment
Equipment
Equipment

RIO Network(optical fiber)

GP-IB device
GP-IB device
Equipment

Equipment

Slave Card

Equipment

Slave Card

Equipment

Slave Card

Equipment

Slave Card

Equipment

Slave Card

Equipment

Equipment

Fig. 2. The VMEbus control system.
3.3 Remote I/O bus
An optical fiber distributed remote I/O (RIO) system is used as the field bus for the VME system. The RIO system wasinitially developed for magnet power-supply control, and eventually will be applied to beam-position monitor andvacuum system control. The RIO system has good electrical isolation and is robust against noise. The system consists of;a) master module with dual port memory, b) several types of slave cards, c) 8-port branch card, and d) star connection
with optical fiber cables between master module and slave cards. The features of RIO system are as follows,- optical fiber connection,- serial link communication,- up to 62 slave cards can be connected to one master module,- 1 Mbps transmission rate,- up to 1 km transmission distance,- HDLC protocol,- can be connected by twisted pair cable (RS-485).
Six types of slave cards are available (type-A, B, D, E, F and G), depending on the type of interface signals of theequipment.
3.4 Operator consoles and servers
Operator consoles consist of seven UNIX workstations. These are HP9000/712/60 workstations running HP-UX with
128MB memory and 2GB disk. Four more consoles are planned to be used as the operator consoles for injectors andmonitoring of human safety and utilities. Consoles are connected to the FDDI network node through an FDDI-Ethernet
switching hub. The GUI based application programs run on each console using the X11 protocol. The connection of
consoles has to be considered carefully because unnecessary transactions on the FDDI backbone may deteriorate its
performance. In our scheme, switching hubs filter transactions efficiently without leaking unnecessary communication

onto the backbone. The secondary database server is an HP J200 which has dual PA-RISC7200 CPUs, 256MB memory,
DAT storage system, CD-ROM juke box and 13GB disk array with RAID mechanism. The primary server will be much
faster than the secondary one.
3.5 Human-machine interface
We are using a commercial GUI builder, X-Mate, for developing the operator interface. The look and feel of X-Mate isbased on Motif 1.2 with X11 protocol. X-Mate does not necessarily require a window manager such as mwm because it
has its own window system on top of Xlib. This is a good point for mission critical 24-hour operation because it is freefrom

Machine operator(s)control panel
GUI

Non-GUI

On-line
Data base

AAAA
AAAA
AAAA
AAAA

Archive
Data baseDiagnostics
Debug
AAAA
AAAA
AAAA
AAAA

Message Server

SYBASESYBASE
Replication

Name
Server

Alarm

AAA
AAA
AAA

AAAA
AAAA
AAAA

AAAA
AAAA
AAAA

AAAA
AAAA
AAAA

AccessServers
AAA

Beam control

AAA
AAAA
AAAA
AAAA
AAAA
AAA
AAAA
AAA
AAAA
AAAA

HP-UX

Collector
AAA
AAA
AAA
AAA

Magnet, RF
Monitor ....

RPC call

RPC call

FDDI Network with TCP/IP

RPC call

HP-RT's
DeviceConfiguration
Table

RPC call
AAAA
AAAA
AAAA

AAA
AAA
AAA

Collector

Equipment Manager

VME module configuration,
SVOC resolve table
etc.

SharedmemoryDevice Drivers
Poller
VME modules

Data or command
Process(es),Store(s)IPC I/F (message)
AAAA
AAAA
AAAA
AAAA
AAAA
AAAA
AAAA
AAAA

Call read-onlyEM

Remote Procedure Call I/F
SQL database accessFig. 3. The software structure of the storage ring control.

the memory leak problem of the Motif window manager. X-Mate provides a good development environment with a
comfortable editor. Users are able to make widgets in WYSIWYG without knowing Motif programming. An interfacepart of the application program has to be written into the call-back routines.

4. Equipment access
4.1 Access scheme
Figure 3 shows the software scheme for equipment access. An operator command is created using the GUI and is sentout to the Message Server (MS), (see 6.1). The MS will forward the message to the Access Server (AS) after checking
its syntax and access control status. The AS parses the message and resolves the destination of the message by asking theName Server (NS). A composite sequence is analyzed in the AS, and messages are sent to the Equipment Manager (EM)running on the remote CPU board. Each equipment group, e.g. Magnet, RF, Vacuum etc., has its own AS. The
synchronization of operations over different equipment is handled here with limited accuracy. The EM is a core process
which manages VME devices connected to equipment controllers (see 4.2).
We have an additional cyclic data-acquisition path to get the equipment status. The Poller process running on the CPU
board sends pre-registered messages to the read-only EM to get the equipment status. The data is stored in memory by
the Poller and taken by the Collector server process on the same board; after collection the data is sent to the Collector
client process on the operator console and finally saved to the on-line database by the DBMS, Sybase.
4.2 Equipment accessWe designed a generic "Equipment Manager" control scheme based on the object-oriented concept [5]. The EM server
process running on the front-end CPU board of the VMEbus realizes a device abstraction concept. It controls accelerator
components directly without knowing the low-level physical device configuration. The operator commands are resolved
by using control semantics and formed into a character string message like an English language syntax S/V/O/C, where
the operator action is translated to a verb (V) and controlled equipment is into an object (O) and so on (see 4.3).
Once the EM receives the abstracted command (message), it is translated from logical equipment to physical device(s)by using the translation table. The translation table consists of a device configuration table and the attribute list ofequipment. These tables keep the hardware addresses of VME modules and calibration constants etc. Tables are createdby an off-line program at the higher level and downloaded to the EM at initialization. After translation, the resolved
information of an operator command is passed to the VME device drivers and finally sent to the front-end equipment.
The response from the equipment is also abstracted and put into the C of the S/V/O/C format. The reply is sent back to
the machine operator through the AS and the MS.
The only way to access the accelerator components is through the EM. The Equipment Manager can be accessed by thePoller locally or by a control application program on the remote consoles by sending a command to the AS in the samemanner.
4.3 Commands
The S/V/O/C control message consists of the subject (S), verb (V), object (O) and complement (C). The subject has
process number, application name, account name, and the host name of the sender which issued the message. The
information of the S is used for access control, exclusion control, and security check. The verb is a control action, theobject is the equipment to be controlled, and the complement is a value to be written. All accelerator components are
abstracted from the physical layer. Programmers of the AS and the Poller are required to know only the abstracted logicalobject (equipment) itself. The details of control channels and status bits are attached to the object as attributes. The
naming scheme for the controlled equipment is chosen to be suitable for beam control at the higher level. Each element
and sub-element of the S/V/O/C message is delimited by the characters "/" and "_". For example, whenever an operator
wants to know the current in a storage ring bending magnet, a character string "get/sr_mag_ps_B/current" is the message
to be sent. The C returned from the EM can be like "123.45A" or "fail".

5. Network5.1 Scheme
As can be seen in Fig. 1, the control system is connected to the duplicated FDDI backbone network with atransmission rate of 100Mbps. There are 6 FDDI nodes for the SPring-8 machine network. One is for the linac and thesynchrotron and the others are for the storage ring. All nodes of the storage ring connnect to the Ethernet/FDDI switchinghub, which provides 10Mbps full bandwidth to each port. We use the optical fiber between the FDDI nodes and the
VMEs in order to avoid magnetic interference. The FDDI optical fiber cables are laid around the maintenance corridor of
the storage ring and to the injector building. The total length of the cable is about 4500m, and almost all the fibers arelaid in a pipe from Air Blown Fiber System (ABFS). The ABFS has multiple pipes inside, and machine control uses one
pipe for a bundle of four fibers.
The primary database server will be in the control room and directly connected to FDDI at that node. We will use thereplication server system on the secondary database server which will interconnect the machine control network and thebeamline user's network. The secondary database server open to users will be a gateway managing access to the control
network by machine users.
5.2 Protocol
We selected TCP/IP for the network communication protocol and Simple Network Management Protocol (SNMP) for
the network management. The TCP/IP protocol is reliable and provides good performance. Reliability is the crucial
feature for machine control. The performance of TCP/IP is acceptable for storage ring control and its architecture issuitable because the configuration of our network is relatively simple.

6. Process communication
6.1 Message distribution
An operator control command is sent by a character string as a message, S/V/O/C as described in 4.1. A message isdelivered to the Access Server through the Message Server by using the UNIX System V message mechanism. The MS
plays a role as a message distributor. The syntax and access privilege of every message passing over the MS are checked
here and all transactions are logged. Each application client running on the workstation has to get a queue number from
the MS. Assigned queues are stored onto the Client Registration Table(CRT) managed by the MS. The CRT is used formonitoring the running clients, and its status can be used for surveillance of clients as well. The message distribution
scheme is designed to allow only one MS on each console. In the first version, there is no communication between MSs
running on different operator consoles; however it may be useful to allow communication between them to enable ordisable (exclusion control) clients on different consoles.
6.2 Remote communication
The Remote Procedure Call (RPC) is selected for communication over the network because it makes the client-server
model more powerful and provides easier programming in the low-level socket interface. There are available commercial
implementations of RPCs such as the Sun's Open Network Computing (ONC) and the OSF's Distributed Computing
Environment (DCE) based on the HP/Apollo NCS RPC. We selected the ONC/RPC because it is more widely available
for the systems which support NFS.

7. Database
7.1 Management scheme
Data for storage ring control are stored in a unified way and accessed by every process running on consoles in theclient-server mode. On-line and off-line applications, distributed in several machines, can store and retrieve the data
to/from the database over the network with remote procedure calls (RPC). We chose a commercial relational databasemanagement system (RDBMS), Sybase SQL server 10.

Sybase was chosen for its on-line performance as well as its capability as a replication server. A replication server
running on server machines can manage the consistency of the data on both the primary and secondary servers. Users ofoff-line applications, or those which do not affect accelerator operation, are only allowed to access the secondary server.
This access control reduces the load on the primary server,which is already heavily loaded by the operational databasemanagement. The replication server system plays an important role as a backup server as well.
7.2 Accelerator database
The storage ring database manages three categories of data, constant parameters and on-line and historical data. The
parameter database stores relatively stable data of equipment, i.e. calibration constants, physical dimensions,relationships between components. These are managed in the normalized database. The data for status of the accelerator
are stored in the on-line database. The Poller/Collector collects current settings and readings of equipment and storesthem into the on-line database. The Poller/Collector system is designed to be fast enough to be able to write about 4000
channels of data, which is approximately 10 KB, to the on-line database once per second. The on-line database isdenormalized to a the flat binary structure to gain performance in updating. The data which are needed for further offline analysis are partially extracted from the on-line database and stored in the historical database with a time stamp. The
archiving cycle of the historical database depends on the data. Alarm status and its history are also written in both the online and historical databases.
7.3 Data access
Data in the database are retrieved by issuing an RPC-based SQL request or by using Sybase client-libraries. The higher
level processes running on consoles can be SQL clients. On the other hand, processes running on the VME CPU boards
can not access the Sybase server directly because that functionality of HP-RT is not available yet. The NFS is one way toaccess text files on a remote node.

8. Status
The design stage of the control hardware has been finished and development is accelerating. The installation of the
VME system for magnets and RF has started in the storage ring. A pipe for the air blown fiber system has been installed
along the SPring-8 accelerators. We have finished the basic design of the control software, and development of some ofthe server processes has started using prototype or spiral prototype methods. The conceptual design of the database has
started recently.
We have installed the prototype control software on the magnet power supplies and confirmed that the basic control
scheme (GUI-MS-AS-EM and Sybase) works successfully on the real system. An operator controlled the power supplies
by using the GUI panel, and the data taken by the EM was stored in the Sybase database.

9. Plans
The beamline control system is expected to be similar to that of the storage ring in hardware and software, but work
has not started yet. The first 10 beamlines, including insertion devices, are planned to be in operation by 1998. It isnecessary to finish the beamline control system soon and to fix a clear methodology for control of the insertion devices
from the machine control point of view. Digital signal processors (DSP) may be useful for the fast local feedback systemfor beam orbit correction and image processing of the X-ray beam shape. Recently, the R & D for the DSP system has
started.

10. Acknowledgments
We would like to express our appreciation to A. Gotz of ESRF, C. Serre of CERN, W. McDowell of ANL, K. Cahill of
FNAL, S. Howry of SLAC, H. Nishimura of LBL, J. Urakawa of KEK and many colleagues of the above laboratories for
useful discussions about the ideas for the control system of the storage ring.

References

[1] M. Hara et al., Proc. 4th European Particle Accelerator Conference, London, June 1994, p597.
[2] T. Wada et al., Proc. of the Int. Conf. on Accelerator and Large Experimental Physics Control Systems,
Tsukuba, Japan, November 1991, p151.
[3] H. Takebe et al., Proc. of the 4th European Particle Accelerator Conf., London, June 1994, p1827.
[4] T. Masuda et al., these Proceedings (ICALEPCS'95, Chicago, USA, 1995).
[5] A. Taketani et al., these Proceedings (ICALEPCS'95, Chicago, USA, 1995).