USING EPICS REDUNDANT IOC IN UNIX ENVIRONMENT*A. Kazakov**, SOKENDAI/KEK, JapanK. Furukawa, KEK, Japan
M. Clausen, G. Liu, DESY, Germany
Abstract

A redundant Experimental Physics and IndustrialControl System (EPICS) input output controller (IOC) isbeing actively developed at Deutsches ElektronenSynchrotron (DESY) in order to achieve high availabilityof control software. The current development focuses on aVME vxWorks environment for cryogenic controls.
However, many scientific facilities use PC-architectureand Unix-like systems such as Linux, Solaris, andDarwin. These facilities require high availability andredundancy. Therefore, this paper describes theimplementation of an EPICS redundant IOC in a PCbased environment with Linux or other Unix-like EPICSsupported OSs. This is achieved by porting a redundancy
monitor task (RMT) and continuous control cxecutive(CCE). The CCE is used to synchronize two RSRV-based
IOC servers. The RMT monitors other parts of thesystem, maintains the connection with the partner, anddecides when to fail-over. It is rather independent andmay be used in a wide range of applications. It has beensuccessfully employed to add redundancy to a CA
gateway.

INTRODUCTION
An EPICS redundant IOC was originally developed atDESY. Two major fields of application were defined:
1. Redundancy for cryogenic plants. In this case, thefailure may be caused by malfunctioning hardware suchas power supplies or fans. An automatic fail-overmechanism should guarantee system stability. However,
over the years it was sometimes necessary to manuallyswitch between the main and backup processors due tomaintenance work during the runtime period (which isusually one year or more). It might be useful for asoftware update. While the current commercial systemused at DESY allows online updates of the database,
EPICS does not allow addition or deletion of records anddatabases during operation.
2.Redundancy for controllers in the XFEL tunnel.
Although the main origin of switch-over in the first casewould be a manual action, it is expected to occur
automatically in the XFEL tunnel. Due to high radiation,
damage to the CPU and memory is highly possible. Thesoftware update is not very important because of more
frequent maintenance days when this operation may beperformed.
By the design draft one major goal was set: Any
redundant implementation must make the system
more reliable than the nonredundant one. Precaution
must be taken especially for the detection of errorsthat shall initiate the fail-over. This operation should
only be activated if there is no doubt that maintaining

the actual mastership definitely causes more damage
to the controlled system than an automatic fail-over.
The fail-over time in any case was defined to be morethan several seconds and less than 15 s. The final
implementation could switch in less than 2 seconds.
Originally, the project was intended to support only
vxWorks and the written code was very specific to it.
However, later it was observed that support for otheroperating system is desirable. Here at KEK we use asoftware-IOC on Linux which functions as a “gateway”
from an old control system to the EPICS-environment. Inaddition, for the ILC project ATCA-based systems under
Linux control will be used. Redundant IOCs are highly
desirable for this project. Thus, the redundant IOC has
been ported to EPICS libCom/osi; this implies that thecurrent implementation should work on any EPICSsupported OSs.

HARDWARE ARCHITECTURE
The hardware architecture consists of two redundantIOCs controlling a remote I/O via shared media such as
the Ethernet or MIL1553. The redundant pair shares twonetwork connections for monitoring the state of health oftheir counterpart, where the private network connection isused to synchronize the backup to the primary and theglobal network is used to communicate data from theprimary to any other network clients requiring the data.

Figure 1: Hardware Architecture

SOFTWARE COMPONENTS
The current design contains three major parts: RMT(redundancy monitoring task), CCE (continuous controlexecutive) and SNL (state notation language) executive.

The latter two are responsible for the synchronization ofprocess variables (PVs) and internal states. The RMT is
the core of the redundant system. It establishes andmaintains the connection with the partner, controls otherparts of the system and decides when to fail-over. Allother components that have to be controlled by the RMTshare the same software interface (which is defined in aheader file rmtDrvIf.h). This interface defines thefollowing functions:1. start: Get access to the IO and start processing.
2. stop: Do not access the IO and stop processing.
3. testIO: Initiates a procedure to test access to theIO.
4. getStatus: Get status of the driver.
5. shutdown: This function is called before the IOC
is rebooted. It terminates transient activities,
deactivates interrupt sources and stops all drivertasks.
6. getUpdate: This routine tells the component to
get an update from the redundant IOC. It is
normally called by the RMT on the inactive IOC.
7. startUpdate: This routine tells the component tostart updating data from the redundant IOC(monitoring). It will first read all fields(depending on the mode) from the redundant
partner. It is normally called by the RMT on theinactive IOC.
8. stopUpdate: This routine tells the component tostop updating data from the redundant
counterpart. This routine is normally called by
the RMT on the inactive IOC.
The RMT can call these functions using an entry tablethat is transferred from the component to the RMT duringinitialization.
Thecomponentcalls
thermtRegisterDriver() function to register itself in the RMTand transfer the entry table.
getInfoCCE and the SNL executive are two such RMTcontrolled components and they implement the RMTdriver interface mentioned above. Other components may
be IO drivers, or any other piece of software. For
example, the RSRV server in redundant IOC
implementation is one of the RMT-controlled components
and it implements the RMT-driver interface. Some of theinterface functions may be unimplemented (set to NULL
in the entry table) depending on the nature of the IOdriver and the tasks it performs.

PORTING
Originally, all the developments waere done only forvxWorks. The resulting code was not usable on any otherOS. However, the people at KEK were interested in using
a redundant IOC on Linux machines for a LINAC controlsystem. Moreover, at DESY there was a demand for aredundant CA gateway. It appeared that the RMT has all
the functionality needed for the solution. However, CA
gateway runs on Linux (or other Unix-like OS), and theRMT was available only on vxWorks. Thus, it wasdecided to port the redundant IOC to Linux.

Porting process
All vxWorks-specific function calls were replaced with
the EPICS libCom/OSI (Operating System Independent
library). In order to accomplish this task the following
new OSI functions were introduced:
epicsMutex.h: epicsMutexLockWithTimeout(id,tmo)
epicsMutex.h:
epicsMutexOsdLockWithTimeout(id, tmo)
epicsThread.h: epicsThreadDelete(id)epicsTime.h: epicsTimeGetTicks ()Further, some modification of the EPICS base weredone to implement CCE hooks and the RMT-driverinterface to the CA server (RSRV), and in database scantasks. These modifications were also made operating
system independent. The following is the list of affectedsource files:
base/src/db/dbAccess.cbase/src/db/dbScan.cbase/src/dbStatic/dbBase.hbase/src/dbStatic/dbLexRoutines.cbase/src/rec/aiRecord.dbd
base/src/rsrv/camsgtask.cbase/src/rsrv/online_notify.cbase/src/rsrv/cast_server.cbase/src/rsrv/caservertask.c
●

●

●
●

●
●
●
●
●
●
●
●
●

Please contact the authors for the patch files.

Results
The ported OSI version of the redundant IOC was
successfully used on vxWorks, Linux, Mac OS X, andSolaris. Tests showed that the system synchronisation
speed limit was around ~5000 records/second for 2 Linuxmachines with 3GHz P4 1core, 2x 100Mbit Ethernet
cards; functioning solely as a redundant IOC.

REDUNDANT CA GATEWAY
One of the demands at DESY was to implement aredundant CA gateway. Besides, after RMT is ported toLinux it fits very well to help this task. CA gateways arebeing widely used in many facilities around the globe
which utilize the EPICS. The CA gateway can be used toadd a security layer or to divide a network into several
segments in order to reduce the amount of broadcast
traffic in each segment. However, when the CA gateway
is utilized, it becomes the “single point of failure” andeven if some or all IOCs are redundant in the case ofgateway failure some clients and IOCs would be cut-off.
Therefore, it seems essential to add redundancy toapplications such as the CA gateway.
By the nature of the gateway task, it has no internal
status needed to be synchronized, and thus, it is desirable
to use both redundant gateway at the same time (loadbalance them). It will reduce the load on each of them andincrease the peak system throughput. Further, in the caseof failure of one of them only half of all the clients will
notice the problem and will reconnect. Hence the task isdivided into two steps.

Redundancy without load-balancing
Redundancy without load-balancing was implemented
by using the RMT as a stand-alone application, whichruns separately from the CA gateway. It provides thebenefit of not modifying the source of the gateway, butraises the problem of how to control the gateway.
Therefore, the chosen solution was to use firewall rules toblock replies from the Gateway process. Only in the caseof load-balancing it is possible to block incoming searchrequests from CA clients. However, when we add loadbalancing it will not work (see next section for details).
In order to add and delete firewall rules, the so called“script-driver” was implemented as the RMT-driver. Upon
becoming a master this driver makes a call to an external“start-script” (which may be any executable file), andupon becoming a slave, it calls an external “stop-script”.
This script driver may be used in other applications andmay call any external application on a specified statuschange. In our case, stop-script adds a firewall rule thatblocks replies from the gateway to the clients, while startscript removes this firewall rules, making it possible tosend replies to the clients. Hence, at any given time there
is only one replying gateway that replies to the clients.
The other gateway is on standby. This scheme worked
well and hence we proceeded to the next step.

address). However, on the slave side, the RMT's "scriptdriver" adds a firewall rule blocking replies. Therefore,
the clients receive only one reply from the master. The
master load-balances the further requests between itself
and its partner. Therefore, after receiving a reply from themaster gateway, the CA client establishes a connection to
one of the two gateways. If we would block the incoming
request from the client instead of the reply from thegateway, the client cannot connect to the slave gateway
later because its list of know PVs would be empty and it
will deny all incoming connections.
As a result of putting all this together, we achieve aload-balancing redundant CA gateway. Some minorchanges to the CA Gateway source code are required.
These changes include signal handling, load-balancing
functionality and new command line options forconfiguring the IP address of the partner and signal
numbers.

Load-balancing
This task appeared to be more tricky. Before we
decided to use RMT as a standalone application, in orderto reduce impact on the original gateway. However, tointroduce load-balancing the RMT needs to have more
control over the gateway. A load-balancing CA gatewayhas to be informed of the current status of its partner. I
our approach of having them as separate processes it wasdecided to use the Unix signals mechanism to inform thegateway process about the status change. Somemodification was done to the gateway source, whichallows the reception of two signals. One of them signifies
that the partner has become “alive”, and the othersignifies that the partner has become “dead”.
The logic of a load-balancing gateway is simple: when
the partner is alive, every other reply from the gateway
includes its partner's IP address (this is the samefunctionality of the CA which is used in the CA directoryservice). When the partner is dead, the Gateway replies
normally. Hence, both the gateways perform the samefunction if they are alive at the same time. Upon receiving
a PV search request from a CA client they perform asearch on the IOC network; if the search is a success, theycreate a “virtual-circuit connection” to the corresponding
IOC. At this stage, the CA gateway adds this PV to the listof known PVs. Subsequently, a reply to the client is sent
(containing either the partner's IP address or it's own IP

Figure 2: Redundant CA gateway

SUMMARY
Porting redundant IOC to Unix-environment allowed amuch wider application of this system. Moreover, it was
shown that the RMT functioning as the core of theredundant system can be utilized to add redundancy toother software.

___________________________________ _________*Work supported by SOKENDAI, Japan**kazakov@gmail.com