Proceedings of ICALEPCS2003, Gyeongju, Korea

EPICS AND WANS: TRADEOFFS BETWEEN ISOLATION, SECURITY,
ROBUSTNESS, AND TRANSPARENCY
J. Hill (LANL), K. Furukawa (KEK), S. Hunt (PSI), A. Johnson (ANL), R. Lange (BESSY),
J. Sage (TJNAF), E. Williams (ORNL)
Abstract
The Experimental Physics and Industrial Control
System (EPICS)[1] was originally designed for use inlocal area networks (LANs). Today, the system isroutinely deployed into complex wide area networks(WANs) using specialized configuration options andproxy gateways. There are advantages to the current
approach including robustness and control over isolationand security. However, some important features are
missing including WAN transparent configuration,
resource location monitoring, detection of name spacecollisions during installation, and wildcard queries into
the resource attribute space. The paper discusses theseissues in detail exploring the relative tradeoffs between
different solutions and including our plans for futureenhancements.

INTRODUCTION
An ideal wide area network (WAN) based control
system would locate resources transparently, notify
clients immediately when a server’s state-of-healthchanges, isolate critical components of the control systemfrom other parts of the system, transparently transfer
loads on critical resources to less loaded systems, andautomatically recover from hardware and software faults.
The following paragraphs introduce some detailed
requirements for the WAN aspects of a control system.
The easier requirements are listed first transitioning into
more difficult to implement requirements.

NAME RESOLUTION ESSENTIALSIn a generic control system clients need to determine
the network address of the server for named resources.
The following is a list of requirements on the nameresolution subsystem for a WAN based and projectgeneric control system.
a Clients will be informed of, and properly respond to,changes in the address of a resources throughout thelifespan of the client.
b Reasonable diagnostics require access to all process
variables and their meta-data. Certain clients will needto resolve thousands of process variable names into
network addresses during initialization. Performanceand efficiency are important.
c Loss of the name resolution subsystem might disrupt
the entire system. Robustness is important.
d It will be possible to integrate servers developed offsite without manually configuring a central authority(plug-and-play capabilities).
e The name resolution load for a large system will growlarger than what any single host can handle and will377

be distributed among multiple hosts as required. It isalso desirable that this load balancing be transparent.
f Clients of the system will not need specialized
configuration to find the location of a resource or tofind the location of the name resolution subsystem.
g The name resolution subsystem will support wildcard
queries into the process variable name space.
h The name resolution subsystem will detect namespace collisions during installation.
i The name space will support hierarchical names, but
the hierarchy shall not impose address boundary rules.
j A self repairing client side name resolution cache will
reduce load on the system.

SERVER STATE-OF-HEALTHNOTIFICATION ESSENTIALS
A publish and subscribe system with notification upon
process variable change of state, and potentially no server
initiated message activity if the process variable does notchange state, requires timely notification when the stateof-health of a server changes. Changes in the server’s
state-of-health might occur when the server is restarted,
the process variable is moved to another server, the paththrough the network is temporarily down, or because ofother hardware or software problems.
a Clients must be notified when a resource isunavailable so that they can enter a fail-safe state.
Proper fail-safe confidence requires that clients must
be continuously notified that a resource they arecommunicating with is operating properly, and that apath to it exists through the network.
b Clients must also be notified when a temporarily
unavailable resource appears or reappears on the
network so that they can immediately connect to it
without loading the network with futile connection
attempts.

ISOLATION ESSENTIALS
It must be possible to configure independent control
system domains which are allowed to interact with eachother only under strictly controlled circumstances. An
isolation barrier between one control system domain andother control system domains enforces the isolation policy
configured by control system integrators.
a An increasing number of clients outside of an
isolation barrier will not result in more than one clientequivalent load inside the isolation barrier.
b The isolation barrier will impose a security policy inaddition to, and overriding, any policy implemented
inside the isolation barrier.

Proceedings of ICALEPCS2003, Gyeongju, Korea

c Other than the security policy that it enforces andcommunication delays, the isolation barrier will betransparent to clients outside of the barrier who must
interface with process variables inside the barrier.
d An isolation barrier will not introduce a single point of
failure for the control system.
e An isolation barrier will not become a bottleneck forthe control system, and therefore will scale withincreasing client load. Automatic load balancing
capabilities are desirable.

CURRENT EPICS PRACTICE
Currently, clients of EPICS determine the network
address of process variables by sending search datagram
messages fully packed with process variable names to alist of server unicast[2] or broadcast addresses. The delay
between search attempts Ρ is based on the estimated client
to server round trip time ε and the lowest unresolved
process variable search attempt countη as follows.

Ρ = 2η ⋅ εThere is also a dynamic adjustment in the number ofdatagrams sent with each search attempt based on past
success rates. After one hundred unsuccessful attempts
for each process variable subsequent search attempts areabandoned.
Server state-of-health notification is communicated bya server beacon datagram sent to a list of client unicast[2]or broadcast addresses. Clients maintain a runningaverage of the period between all server beacons received.
A missing beacon from a server to which the client isconnected results in the client manually verifying that
virtual circuit with a loop back message. A substantialchange in any server beacon results in a new server eventin the client. This resets the search attempt count to nohigher than six for each unresolved process variabletypically resulting in a new initial search period of sixty
four times ε .

EPICS BROADCASTING ISSUES
Broadcast messages are the network bandwidth
efficient way to send a copy of an identical message tomultiple hosts. IP broadcast addresses can be used toreach any server throughout the client’s subnet and netdirected broadcasts[2] can be used to reach any server ona remote subnet as router configuration permits. Internet
multicast addresses can be used to reach any server on theInternet that has registered interest in the specified
multicast group id, independent of router configuration.
EPICS could be easily modified to support multicasting.
Broadcast based protocols are common. For example theinternet protocols ARP, DHCP, the X window system’s
XDMCP can be configured to utilize hardware
broadcasting for the purpose of locating resources. There
are many preconceptions for and against hardware
broadcasting and therefore the next few paragraphs are a
discussion of the positive and negative aspects of thebroadcast based protocols in EPICS.
378

The advantages of broadcast based server state-ofhealth beacons are efficient use of network bandwidthcompared to periodic state-of-health heartbeats over a
virtual circuit, and improved server communication loss
event detect confidence compared to schemes using acentralized server state-of-health authority. The
centralized scheme is oblivious to localized network
failures and it also introduces additional links in theserver loss detect logic introducing additional failure
scenarios, and therefore decreased confidence.
A clear disadvantage of broadcast based server state-ofhealth beacons in large systems with multiple subnets isthat proper configuration can become tedious. Proper
client side detection of newly available servers requiresthat all clients see beacons from all available servers, andproper configuration ensuring this can be daunting for
EPICS system managers, but this negative could be
eliminated if EPICS used IP multicasting. Beacon period
estimation errors induced by client load bursts, server
load bursts, or aberrant network segments can result infalse new server events (see below).
The advantages of broadcast based name resolution
include no need to install and maintain a centralized nameserver, no single point of failure, and integration ofautonomous offsite development without manuallyconfiguring a central name service authority. EPICS iscapable of over 15k virtual channel connects per secondon 2001 vintage PC hardware. This figure includes thedatagram based name resolution phase, and also and thevirtual circuit connect phase for each channel so inpractice EPICS appears to meet requirement 1b.
The disadvantages of broadcast based name resolution
include increased configuration complexity for off IP
subnet clients that is especially problematic if routers are
not configured to accept net-directed broadcasts[3], but
this negative could be surmounted if EPICS weremodified to support IP multicasting. Within long lifespancontrol systems the typical propensity is to end up with amodern workstation CPU, a modern LAN switch, and alegacy front end controller CPU. In this situation thename resolution efforts of the workstations tends to
produce a more significant load on the front end
controller CPU. Addition of a name resolution cache in
the client would reduce traffic, but wouldn’t resolve theprimary problem: there is a tendency towards growing
numbers of dispossessed process variable names in theclient configurations of large systems. Exponential backoff in the client’s search rate tends to moderate thisproblem, but false new server events detected in theclients resulting from client load bursts, server load
bursts, or aberrant network segments can be problematic.
Of particular concern might be a self amplifying situationin large EPICS systems, where increased levels of nameresolution related broadcasting activity resulted in server
load bursts. The quiescent network and CPU loads
resulting from this phenomenon tend to be quite low, but
the peak loads have not been well characterized. Work isunderway to allow these metrics to be archived over time.
It is known that the software is designed to systematicallyProceedings of ICALEPCS2003, Gyeongju, Korea

degrade when peak loads reach saturation. For example,
in the server, the priority of the search message input
thread is just below the beacon generating thread which isjust below the threads servicing virtual circuits. This
ensures that search message traffic does not disrupt
regular beacons, but a subscription update load carriedover the virtual circuits which saturates the CPU will
disrupt the beacons and result in the clients detecting thatthe server is no longer responsive. While this is correct
per-design behaviour it is also expected that any server inthe system with an intermittently saturated CPU willproduce intermittent beacons, and that could lead to false
new server events in the clients.

LARGE EPICS INSTALLATIONSThere are a number of options available to large EPICS
installations for managing the above issues. Several sites
have used a server side plug-compatible name resolution
interface to implement alternative name resolution
services allowing clients to find their resources without
issuing name resolution broadcasts. The name server usedby the CEBAF control system at Jefferson National
Laboratory is frequently used as a starting point for site
specific name resolution services[3]. These solutions
generally work well, but they introduce a single point of
failure (violating requirement 1c) and do notcommunicate state-of-health information. An alternative
solution with strong architectural merits employs the
EPICS gateway to implement isolation barriers breaking
up a large EPICS system into a loosely coupled group oflightly loaded subsystems, but this approach currently
also introduces geographically limited single points offailure (violating requirement 1c). Use of EPICS
gateways and alternative name services also reduce theconfiguration effort required for clients to communicate
with off subnet servers.

POTENTIAL EPICS IMPROVEMENTS
The default EPICS system today only meetsrequirements 1a through 1e, 2a through 2b, and 3a
through 3c. Some upgrades are required to meet
additional requirements. Modifying the EPICS nameresolution subsystem to support wild card queries(requirements 1g) and to immediately detect name spacecollisions (requirement 1h) will require a centralized
name resolution authority. Some searching on the Internet
indicates that the feature sets in DNS[4] and LDAP[5] are
candidates for implementing requirements 1a though 1j.
Some initial performance measurements for these systems
indicate name resolution throughput between 10 and a
100 times slower than existing EPICS mechanisms(requirement 1b). This might not be problematic for manysystems, but for EPICS which employs a particularly finegrained resource naming granularity the choice must be

379

made carefully. Performance issues will be most likely toarise when clients connect large numbers of virtualchannels and when initializing servers upload theirresource name lists to the central name authority(requirement 1d). A potential solution might be to
implement an alternative performance oriented client
library for these systems allowing multiple nameresolution requests to be sent per socket IO call, but thismight be unnecessary considering that both LDAP andDNS have caching capabilities. The content
synchronization extensions [6][7] recently proposed for
LDAP are interesting because they may address
requirement 2b. They would not however provide
appropriate fail safe behaviour (requirement 2a) because
proper fail safe behaviour and proper detection ofpotential hardware and software faults requires directcommunication with the server. Redundancy and load
distribution capabilities built into these systems probably
do address requirements 1c and 1e.
Solutions for requirements 3d and 3e may also be
coupled with our choices for the name resolution system.
If the name resolution system informs the clients toswitch to an alternative EPICS gateway (isolation barrier)whenever the primary gateway has failed, or isoverloaded, then perhaps requirements 3d and 3e can besatisfied by running redundant gateways which operate inparallel. This approach might also be useful for
implementing redundant EPICS input output controllers.

CONCLUSIONSThe paper provides a list of WAN requirements for ageneral purpose control systems, discusses the limitations
present today related to installation of large EPICS
system on WANs, and examines the components that are
currently available for coping with these limitations. Ourplans concerning new features including wildcard queries,
resource name space collision detection during
installation, easier configuration, more robust operation,
and EPICS systems transparently available throughout theinterconnected internet were also discussed.

REFERENCES[1] http://epics.aps.anl.gov/[2] “TCP/IP Illustrated, Volume 1”, W. Richard Stevens,
Chapter 1.4, Addison Wesley[3] J. Sage, “Using a Name Server to Enhance Control
System Efficiency”, ICALEPCS’01, San Jose,
November 2001.
[4] http://www.dns.net/dnsrd/[5] http://www.openldap.org/[6] http://www.openldap.org/pub/kurt/ldapsync.ppt[7] http://www.ietf.org/internet-drafts/draft-zeilenga-ldupsync-03.txt