HIGHER AVAILABILITY WITH ATCA AND REDUNDANT IOC○ A.Kazakov1,2,K.Furukawa1, M.Satoh1, T.Suwada1, S.Michizono1
1高エネルギー加速器研究機構(KEK)、2総合研究大学院大学(SOKENDAI)USING EPICS REDUNDANT IOC IN UNIX ENVIRONMENT*
Abstract

A. Kazakov**, SOKENDAI/KEK, Japan

Reliability issues for modern accelerator machines become more and more important due to constantly growing andK.forFurukawa,Japan
complexity of the machines. Redundant IOC
EPICS wasKEK,
introduced
in 2006 and ported to Linux in 2007. Redundant
G. Liu,Germany
IOC is aimed to provide redundancyM.
andClausen,
high availability
forDESY,
critical partsof an accelerator control system. The core ofEPICS redundant IOC is Redundancy Monitoring Task (RMT). In this work Advanced Telecommunication Computing
theredundantactual mastershipdefinitely
causesmore
Abstract
Architecture (ATCA) driver to RMT (and therefore to EPICSIOC) is being
developed.
ATCA
is damagea populartothecontrolledsystemthananautomatic
fail-over.
AredundantExperimental
PhysicsandIndustrial
standard for new designs and installations of control systems. ATCA was suggested as a base architectureforThe fail-over time in any case was defined to be more
Control
SystemLinear(EPICS)
input (ILC)
outputcontrolcontroller(IOC)This
is paperInternationalCollidersystem.
describes the implementation of an EPICS redundant
than several seconds and less than 15 s. The final
being
activelyin developed
Deutscheswith
ElektronenIOC (RIOC)
a PC-based atenvironment
Linux or other Unix-like EPICS-supported OS’s and benefits of using
Synchrotron (DESY) in order to achieve high availability implementation could switch in less than 2 seconds.
ATCA as platform for RIOC.
Originally, the project was intended to support onlyof control software. The current development focuses on avxWorks
and the written code was very specific to it.
VME vxWorks environment for cryogenic controls.
However,later it was observed that support for other
However, many scientific facilities use PC-architecture
operatingsystemand
Unix-likesystems
suchasLinux,Solaris,andoperating
systemisisdesirable.
desirable.Here
HereatatKEK
KEKwe
weuse
usea a1. Introduction
Darwin. These facilities require high availability and software-IOCsoftware-IOCononLinuxLinuxwhich
whichfunctions
functionsasasa a“gateway”“gateway”
An EPICSTherefore,redundant IOC
originallydevelopedredundancy.
thiswaspaper
describes
theat fromfromananold
oldcontrol
controlsystemsystemtotothe
theEPICS-environment.
EPICS-environment.InInDESY.
Twomajor
fieldsofapplication
weredefined:
addition,fortheILC
project
ATCA-based
implementation of an EPICS redundant IOC in a PCaddition, for the ILC project ATCA-basedsystems
systemsunder
underRedundancywith
for cryogenicplants.
In this case,the Linux control will be used. Redundant IOCs are highly
based1.environment
Linux or other
Unix-like
EPICSLinux control will be used. Redundant IOCs are highly
failure maybeThis
causedby malfunctioning
such desirable for this project. Thus, the redundant IOC has
supportedOSs.
is achievedby porting ahardware
redundancy
desirable for this project. Thus, the redundant IOC has
as poweror fans.
An automatic
fail-over been ported to EPICS libCom/osi; this implies that themonitor
task supplies(RMT) andcontinuous
control cxecutive
been ported to EPICS libCom/osi; this implies that themechanism
shouldguarantee
system stability.
However, current implementation should work on any EPICS(CCE).
The CCE
is usedto synchronize
two RSRV-based
current implementation should work on any EPICSOSs.
over servers.
the yearsThe
it wassometimes
to manually
IOCRMT
monitorsnecessary
other partsof the supported
supported OSs.
switch maintains
between theand backup
system,themainconnection
with processors
the partner,due
andto
maintenanceduring Ittheis runtime
period (whichdecides
when work
to fail-over.
rather independent
andis
usually
one inyear
or rangemore).ofItapplications.
might be usefulfor a
2. Hardware
Architecturemay
be useda wide
It has been
HARDWARE
ARCHITECTURE
successfullyemployed
redundancy
to asystemCAsoftware update.
Whiletotheadd
current
commercialThe
Thehardware
hardwarearchitecturearchitectureconsists
consistsofoftwo
tworedundant
redundant
gateway.
used at DESY allows online updates of the database, IOCs
IOCscontrolling
controllinga aremoteremoteI/O
I/Oviaviashared
sharedmedia
mediasuch
suchasas
EPICS does not allow addition or deletion of records theEthernet
orMIL1553.
Theredundantpairshares
the Ethernet or MIL1553. The redundant pair sharestwo
twoand databases during operation.
network
networkconnections
connectionsfor
formonitoring
monitoringthe
thestatestateofofhealthhealthofof
2.Redundancy
for controllers in the XFEL tunnel. their
INTRODUCTION
theircounterpart,
counterpart,wherewherethe
theprivate
privatenetwork
networkconnection
connectionisis
Althoughthemainoriginswitch-overthe first case
An EPICS redundant
IOCofwasoriginallyindeveloped
at usedusedtotosynchronize
synchronizethe
thebackup
backuptotothe
theprimaryprimaryand
andthethewouldbeamanualaction,it
isexpected
tooccur
globalnetwork
isusedtocommunicate
datafromDESY. Two major fields of application were defined:
global network is used to communicate data fromthe
theautomaticallyin thetunnel.
Due Into thishigh case,radiation,othernetworkclients
requiringthe data.
1. Redundancy
for XFEL
cryogenic
plants.
the primaryprimarytotoany
anyothernetwork
clients
requiringthe data.
damageto be
the caused
CPU andis highly
possible.such
Thefailuremayby memory
malfunctioning
hardwareis notbecausefail-overof moreassoftware
power update
supplies
or veryfans. important
An automatic
frequent maintenancedays whenthisstability.
operation
may bemechanism
should guarantee
systemHowever,performed.
over the years it was sometimes necessary to manuallyswitch between the main and backup processors due to
By the design
draft onemajor goalwas(whichset: Anymaintenancework during
the runtime
periodisredundant
implementation
the system
usually
one yearor more). It must
mightmakebe usefulfor asoftwareupdate.thanWhilecurrent commercialsystem
more reliable
the thenonredundant
one. Precaution
usedonlineforupdatesof the database,mustatbeDESY
takenallows
especiallythe detectionof errors
EPICS
does initiate
not allowtheaddition
or deletionof records
andthat shallfail-over.
This operation
shoulddatabasesduringoperation.
only be activated if there is no doubt that maintaining
2.Redundancy
for controllers
the XFEL
tunnel.
theactual mastershipdefinitelyincauses
more damage
Althoughthe main origin
switch-overin the firstcase
to the controlled
systemofthan
an automatic
fail-over.
would
be a manualis expected
The fail-overtime in action,
any caseit wasdefined totobeoccur
more
automaticallythe XFEL
than several inseconds
andtunnel.
less Duethan to15highs. radiation,
The finaldamage
to the CPU
andswitchmemoryis highly
possible. The
implementation
could
in lessthan 2 seconds.
software update is not very important because of more
Figure 1: RIOC Hardware Architecture
frequent maintenance days when this operation may be
Originally, the project was intended to support only Figure 1: Hardware Architecture
performed.
vxWorks
and the draft
writtenverywasspecificto it.
By the design
one codemajorwasgoal
set: Any
However, later
it was observed
supportother
redundant
implementation
mustthat
makethe forsystemSOFTWARE COMPONENTS
more reliable than the nonredundant one. Precaution
must be taken especially for the detection of errors
The current design contains three major parts: RMT
that shall initiate the fail-over. This operation should (redundancy monitoring task), CCE (continuous control
only be activated if there is no doubt that maintaining executive) and SNL (state notation language) executive.

3. Software Components
The current design contains three major parts: RMT(redundancy monitoring task), CCE (continuous controlexecutive) and SNL (state notation language) executive.

scan task

ca servr

Normal IOC

RMTRMT
Driver r
Drive

RMT

rmtRegisterDriver() function to register itself in the RMTand transfer the entry table.
CCE and the SNL executive are two such RMTcontrolled components and they implement the RMTdriver interface mentioned above. Other components may
be IO drivers, or any other piece of software. For
example, the RSRV server in redundant IOC
implementation is one of the RMT-controlled components
and it implements the RMT-driver interface. Some of theinterface functions may be unimplemented (set to NULLin the entry table) depending on the nature of the IOdriver and the tasks it performs.

4. Porting to libCom/OSICCERMTDriverFigure 2: RIOC internals
The latter two are responsible for the synchronization ofprocess variables (PVs) and internal states. The RMT is
the core of the redundant system. It establishes andmaintains the connection with the partner, controls otherparts of the system and decides when to fail-over. Allother components that have to be controlled by the RMTshare the same software interface (which is defined in aheader file rmtDrvIf.h). This interface defines thefollowing functions:
1. start: Get access to the IO and start processing.
2. stop: Do not access the IO and stop processing.
3. testIO: Initiates a procedure to test access to the IO.
4. getStatus: Get status of the driver.
5. shutdown: This function is called before the IOC isrebooted. It terminates transient activities,
deactivates interrupt sources and stops all drivertasks.
6. getUpdate: This routine tells the component to get an
update from the redundant IOC. It is normallycalled by the RMT on the inactive IOC.
7. startUpdate: This routine tells the component to start
updating data from the redundant IOC(monitoring). It will first read all fields(depending on the mode) from the redundant
partner. It is normally called by the RMT on theinactive IOC.
8. stopUpdate: This routine tells the component to stop
updating data from the redundant counterpart.
This routine is normally called by the RMT on
the inactive IOC.
The RMT can call these functions using an entry tablethat is transferred from the component to the RMT duringinitialization.
The
component
calls
theOriginally, all the developments were done only for
vxWorks. The resulting code was not usable on any other
OS. However, the people at KEK were interested in using
a redundant IOC on Linux machines for a LINAC controlsystem. Moreover, at DESY there was a demand for aredundant CA gateway. It appeared that the RMT has all
the functionality needed for the solution. However, CA
gateway runs on Linux (or other Unix-like OS), and theRMT was available only on vxWorks. Thus, it wasdecided to port the redundant IOC to Linux.
The ported OSI version of the redundant IOC wassuccessfully used on vxWorks, Linux, Mac OS X, andSolaris. Tests showed that the system synchronisation
speed limit was around ~5000 records/second for 2 Linuxmachines with 3GHz P4 1core, 2x 100Mbit Ethernet
cards; functioning solely as a redundant IOC

5. AdvancedTCA
Advanced
Telecommunications
ComputingArchitecture (ATCA) is a new standard in telecomindustry. Originally ATCA specification was designedwith high availability in mind to be used in wide
variety of applications requiring higher availability.
Every ATCA shelf consists of the following majorparts[1]:1. Subrack providing attachment points for theBackplane, as well as alignment, support, andmechanical engagement for the insertion andextraction of Front Boards and RTMs.
2. Backplane providing connector interfaces for
power
distribution
andinput/output
connectivity between Front Boards.
3. Rear Transition Module (RTM) installed in therear part of the shelf and mated to Front Boards.
4. Front Boards containing the desired electronic
functions and the connectors required to interface
with these functions.
AdvancedTCA provides for extensive management
capabilities, which may be used by the overall SystemManager. The Shelf Management system does thefollowing:• Monitors, controls, and assures proper operation ofAdvancedTCA Boards and other Shelf components• Watches over the basic health of the system, reports
anomalies, and takes corrective action when needed• Retrieves inventory information and sensor readings
as well as receive event reports and failure notifications
from Boards and other Intelligent FRUs
• Performs basic recovery operations such as power
cycle or reset of managed entities• Provides low-level hardware management services tomanage the power, cooling, and interconnect resourcesof a Shelf
One of the key components of ATCA Shelf
Managements system is Shelf Manager board (preferablyredundant). The Shelf Manager does the following:• Watches over managed devices, reporting anomalous
conditions to the System Manager and taking whatevercorrective actions it can to prevent system failure• Handles hot-swap events from removable devices,
indicating their entry into the Shelf and detecting their
shutdown or removal• Negotiates power budgets with Boards and other
FRUs so that the Shelf operates within those capacities.
• Initiates changes in fan levels when event messages
show that temperatures are outside of prescribedbounds.
The management is realized via Intelligent Platform
Management Bus (IPMB), which is a part of Intelligent
Platform Management Interface (IMPI) architecture.

Platform Interface specification separates the hardware
from management middleware and makes eachindependent of the other. Therefore, an application
developed using HPI should be independent fromparticular hardware realization.
Using OpenHPI library an RMT-driver for ATCA isgoing to be implemented. Via this driver RMT can get
information about health status of the partner board andthe shelf itself. This information can give an ability to
predict the failure and initiate a failover process before
actual hardware starts to fail. For example if for somereason the temperature on the master board starts to raise,
RMT could switch over to a slave board when the masterboard is still running. Therefore the failover happens inmore stable and controlled environment. Because thehardware is not failing channel access connections could
be gracefully closed and clients would initiate thereconnect procedure immediately after switchover toslave IOC. In a matter of a second all the connections
would be restored and total control is regained. In case of
typical scenario when the connections are not closed
gracefully it takes 30 seconds for clients to reconnect toslave IOC.

BladeRMT

HPI-Library

ATCA platform uses redundancy internally for
management system connections, shelf managers,
backplane interconnections, network subsystem.
Therefore it is a good basis for critical applications inaccelerator control systems. ATCA was chosen as astandard for future ILC control system. But justrunning EPICS IOC’s on the ATCA-standard CPU
board does not make it redundant, even if there isanother identical board. To solve this problem
Redundant IOC should be utilized.
Ported to OSI library version of RIOC wassuccessfully tested to run in ATCA-standard shelf.
This configuration provides EPICS redundancy, whichis essential for critical tasks and based on highly
reliable ATCA standard.

7. ATCA driver for RMT and RIOC
Running RIOC in ATCA shelf is great in the sense ofdelivering better availability, but it does not utilize any of
ATCA management features available.
A lot of management and health information can beretrieved via Shelf Manager board. This information canbe acquired using Hardware Platform Interface. Hardware

...

IPMC

IPMC

...

Blade2 sensors

IPMC

IPMC

FANS
IPMC

Blade1 sensors

EPICS IOCip6. Advanced TCA and RIOC...

Shelf-manager
HPI Daemon SNMP Agent
Self-management controller

IPMC

Redundant IPMB
Figure 3: RIOC and ATCA driver

Summary
Porting redundant IOC to Unix-environment alloweda much wider application of this system. It was shownthat the RMT functioning as the core of the redundant
system can be utilized to add redundancy to othersoftware. And it can be used to build highly reliable
redundantsystemsbasedonATCA-standard.
Development of RMT driver to support ATCAmanagement system will deliver even better stability andperformance.
[1] AdvancedTCA, PCIMG 3.0 Short Form Specification,January 2003.