An Accelerator Controls Network Designed for Reliability and Flexibility*William P. McDowell and Kenneth V. Sidorowicz
Advanced Photon Source, Argonne National Laboratory, 9700 Cass Ave, Argonne, IL USA
Abstract
The APS accelerator control system is a typical modern
system based on the standard control system model, which
consists of operator interfaces to a network and computercontrolled interfaces to hardware. The network provides a
generalized communication path between the hostcomputers, operator workstations, input/output crates, andother hardware that comprise the control system. Thenetwork is an integral part of all modern control systems
and network performance will determine many
characteristics of a control system. This paper describes
the methods used to provide redundancy for various
network system components as well as methods used toprovide comprehensive monitoring of this network. The
effect of archiving tens of thousands of data points on aregular basis and the effect on the controls network will bediscussed. Metrics are provided on the performance of thesystem under various conditions.

vendors. The TCP/IP implementation is independent of theparticular network medium selected to implement thenetwork. APS uses FDDI, 10-Mbit Ethernet and 100-MbEthernet.
The APS user community has very high expectations forthe guaranteed delivery of beam. The goal for APS
operations is to provide greater than 90% availability(actual hours/scheduled hours). Four percent of thescheduled time is required to fill the storage ring, whichleaves only 6% of the availability budget for all accelerator
systems failures. The downtime budget for the control
system is less than 0.05%. During the operational period
from 8/19/97 to 9/15/97 beam was delivered for 495 of560 scheduled hours, for an overall availability of 88.3%.
The error budget for the control system would have been
2.8 hours. In fact, the control system accumulated 0.0
hours of downtime during this period. One of the factors
behind the high reliability of the control system has been
the excellent reliability of the network.

1 Introduction

2 Network overview

The APS accelerator control system has been
implemented using the Experimental Physics andIndustrial Control System (EPICS) software tool kit. At
the APS, the operator interface is a SUN Microsystems
UNIX workstation with an X-windows graphical userinterface. As is common to this type of system, an operator
workstation or X-terminal may be located at any physicallocation in the facility. An operator has the ability togenerate and alter control displays and to accessapplications, interactive control programs, custom code,and other tools. The front-end computer or input/output
controller (IOC) provides direct control and input/output
interfaces for each accelerator subsystem. At APS thestandard crate uses the VME or VXI bus standard, aMotorola 68040/60 processor, Ethernet-based network
communications, and a variety of signal and sub-network
interfaces. The 68040/60 processor provides the crate with
the intelligence to allow it to run its software
autonomously with respect to all other devices in thesystem. The EPICS core software running in the cratehides hardware dependencies from the high-level software
running on the workstation. There are approximately 175
crates used in the accelerator control system. A real-time
operating system, VxWorks, is run in the crate central
processing unit (CPU) to provide the basis for the realtime control.
EPICS uses the TCP/IP networking protocol, acommercial standard supported by all network hardware

Figure 1 gives an overview of the complete APS
computer network including that portion of the networkthat is used to control the accelerators. The accelerator
control network uses optical fiber to connect satellite
network hubs to a collapsed backbone FDDI concentrator
system. All the hubs are dual attached to the concentrator
using a star connection configuration. The APS network
must serve several diverse functions including accelerator
control, beamline control, experimental data acquisition,and normal day-to-day computerized office functions suchas word processing. A router is used to isolate the APS
network functions. The network lends itself to being
divided along geographical lines, and thus the network isdivided into the control system, the CAT beamlines andlaboratory office modules (LOMs), the central laboratory
office building (CLO), and the Argonne Guest House(AGH).
The network hub equipment and cabling plant allows anupgrade path to fast Ethernet, Gigabit Ethernet, or ATM
technology if future needs show that an upgrade is needed.
In addition, all connections and equipment will allow failover to redundant paths and equipment.
The same equipment and strategy has been followed forthe network hub equipment in the CLO, the CAT areas,and the AGH buildings. Thus the computers installed inoffices, labs, and residence rooms will use Category 5wiring at 10 or 100 Mbit/sec Ethernet rates, while thenetwork equipment will be able to be upgraded to ATM or*

Work supported by U.S. Department of Energy, Office of Basic Energy Sciences under Contract No. W-31-109-ENG-38.

Gigabit Ethernet on an incremental basis.

Figure 1.

module in the hub is completely independent sharing onlya passive backplane.

APS network.

3 The accelerator control network
A block diagram of the APS control system hub networkis given in Figure 2. An FDDI output card in the APS
router is attached to one of the collapsed backbone FDDI
buses on the Cabletron MMAC Plus enterprise hub. Thishub was selected to provide high availability network
services to the APS control system. The MMAC Plus hub
accommodates 14 interface modules and has fault tolerant
features built into it. It has two dual FDDI networks which
provide up to 400 Mbps of network bandwidth, and it will
support both packet and ATM cell transport.
There are ten remote hubs in the controls network
distributed throughout the accelerator facility in order toprovide local Ethernet connections to all network devices.
These remote hubs are dual attached to the central
MMAC+ hub using FDDI in a star connection
configuration.
This allows network reconfiguration
without the addition of a new fiber plant if futuretechnology, such as Gigabit Ethernet or ATM, is installed.
The system also uses different physical paths for the fibers
between the remote hubs and the concentrators in order toprovide protection against common mode physical damage.
There are four hubs serving the storage ring, two serving
the rf system, two serving the injector system, and twoserving the systems in the main control room (MCR). Eachremote hub is attached to both concentrators and eachconcentrator
iscompletelyindependent.
This
independence was one of the primary reasons a Cabletron
MMAC+ hub was selected for this application. There is nocommon management module in the MMAC+; each

Figure 2.

APS controls network.

In addition, two independent, redundant power supplies
power the MMAC+ hub. Thus it would take multiple
independent failures to cut communication with the remotehubs. All of the control system IOCs are connected to thehubs using fiber Ethernet and they too can be reconfigured
if required. To provide redundant service, every IOC isconnected to two hubs using a redundant channel fiber
optic Ethernet transceiver. Each transceiver is connected
to two hubs: a primary hub with fibers running clockwisearound the storage ring from the IOC to the hub and asecondary hub with fibers running counterclockwisearound the storage ring from the IOC to the hub. Thistransceiver has two ways of determining that the network
has changed. The first is to monitor the link integrity pulse.
If the transceiver senses that the link has been lost, it willautomatically switch over to its backup port and reflectthat change in the LEDs located on the front of the unit.
The other method to determine network status is to sense
the presence of data on one port of the fiber optic ports.
The first port to sense data will be designated as the activeport. The transceiver can be configured for either a 2-sec
or a on 10-sec quiet period. If the transceiver does notdetect data the primary port for the selected time period,

the unit will automatically switch to its backup port. Whenthe switch is made, LEDs on the front panel indicate thebackup port is now the primary port. This allows a hub tobe serviced or to fail without causing the IOC to lose
communication with the network. A block diagram of atypical remote hub is shown in Figure 3.

4 Conclusion
Network reliability can be defined as the ability of thenetwork to remain operational (available) when some ofits components have failed. Failure states in the network
are caused by device failures, such as transceiver chatter
and/or link failures. In general, these equipment failures
are assumed to be statistically independent. At the APS wehave provided a great deal of protection against singlefailures. In our experience, hardware failure has caused
95% of network crashes. To provide the reliability
demanded by the APS users, we have provided a networkwith sufficient redundancy to meet the requirement of99.95% availability.

HUBSR32-ETHERNET
Start: June 12 – 9:00am
Stop: June 17 – 8:00am

Figure 3. Typical remote hub.
A typical remote hub contains six 12 port fiber Ethernet
modules, each of which is connected to an Ethernet switchthat in turn connects to the concentrators via FDDI. Eachhub module supports six primary IOC ports and six
secondary IOC ports. Thus under normal conditions only
six IOCs share an Ethernet segment.
A sniffer plot of a heavily used IOC (the main BPM IOC)is given in Figure 4. To obtain these plots APS has
installed a distributed sniffer system that allows any
Ethernet segment (six per hub) or an FDDI segment to be
remotely selected and monitored by a hardware sniffer.
The sniffer software supports the X-windows protocol andtherefore can be viewed on any X-enabled workstation orPC. Support staff can monitor and diagnose network
problems from their office or their home PC using theAPS-provided ISDN network link. This allows the system
to be diagnosed and repaired without requiring the support
staff personnel to travel to the Laboratory during off-hours.
There are twelve workstations in the main control room:
seven of these are used by operations to control various
aspects of the facility, and five of the workstations areavailable for general use by engineering groups. Eachgroup of six workstations is single-attached by FDDI toeach concentrator so that a concentrator failure will not
impact operations.
There are also two file servers in the control system.
These computers are dual-attached to each concentrator.
Six workstations boot from each server to prevent thecontrol system from becoming disabled in the event of aserver crash.

IOC’s  iocs31-39vp (Primary)
iocs21-29vp (Secondary)

S35 POWER SUPPLYPROBLEMS

Figure 4.

S20 POWER SUPPLYPROBLEMS

Sniffer plot.