Control System Reliability at Jefferson Lab
Karen S. White, Hari Areti, Omar Garza
Thomas Jefferson National Accelerator Facility
12000 Jefferson Ave., Newport News, VA 23606
USAAbstract
At Thomas Jefferson National Accelerator Facility(Jefferson Lab), the availability of the control system iscrucial to the operation of the accelerator for experimentalprograms. Jefferson Lab's control system, uses 68040
based microprocessors running VxWorks, Unix
workstations, and a variety of VME, CAMAC. GPIB. andserial devices. The software consists of control system
toolkit software, commercial packages, and over 200
custom and generic applications, some of which are highly
complex. The challenge is to keep this highly diverse andstill growing system, with over 162,000 control points,
operating reliably, while managing changes and upgrades
to both the hardware and software. Downtime attributable
to the control system includes the time to troubleshoot andrepair problems and the time to restore the machine tooperation of the scheduled program. This paper describes
the availability of the control system during the last year,
the heaviest contributors to downtime and the response toproblems. Strategies for improving the robustness of thecontrol system are detailed and include changes inhardware, software, procedures and processes.
Theimprovements range from the routine preventive hardware
maintenance, to improving our ability to detect, predictand prevent problems. This paper also describes thesoftware tools used to assist in control system
troubleshooting, maintenance and failure recovery
processes.
1 Introduction
The Continuous Electron Beam Accelerator at Jefferson
Lab provides high current electron beams of up to 4 GeV
energy to three experimental halls. The machine consistsof two superconducting linear accelerators connected
together with 9 arcs. An injector system provides polarized
and unpolarized electrons from two different sources. RFand magnetic separation in the beam switch yard splits andtransports beam to the halls. The range of deliverable
currents to the halls spans <1 nanoAmp to 120 microAmps
and requires a broad class of diagnostic equipment. The
linacs contain 340 superconducting cavities and theaccelerator has well over 2000 magnets. Additionally, themachine utilizes a variety of diagnostic devices for beam
delivery.
The task of the control system is to provide control andmonitoring for all these elements as well as various
graphical user interfaces, archival of data, alarms andanalysis tools for machine operators, engineers andphysicists. The goal is to achieve 98% control system
availability during accelerator operations in support of ourgoal of overall machine availability of 70%.
2 Control system
The Jefferson Lab Control System, which is built on theExperimental Physics and Industrial Control System(EPICS), is a distributed system using a client-server
architecture, name based I/O and TCP/IP communication[1]. The system consists of two computing levels. The firstlevel is composed of Unix workstations and X-terminals
which execute a wide variety of system applications suchas operator interfaces, archiving, backup and restore,alarms, and downtime logging. Also at this level, highlevel applications, for automated machine setup andcontrol, are executed and displayed. The second level iscomposed of VME 68040 based single board computers
running the VxWorks real-time operating system, EPICS
system software and the corresponding device control
applications. These computers are called Input/Output
Controllers (IOCs) and connect to various types of
hardware control modules. Some of the IOCs also
communicate with embedded microprocessors for
specialized device control. The computers at the two levels
communicate over a segmented ethernet network.
The EPICS IOCs are connected to hardware modules
through a variety of interfaces including CAMAC, VME,GPIB, Serial and some custom interfaces. The mostcommon type of hardware interface used for control at
Jefferson Lab is CAMAC. Over 100 CAMAC crates
containing more than 1000 CAMAC modules
communicate with the IOCs via a Serial Highway using
Hytec Serial Highway Driver cards and L2 Serial Crate
Controllers. More than 100 VME modules used for device
control are located in the same VME enclosures whichhold the IOCs. About a dozen GPIB devices are interfaced
to EPICS IOCs using a second 68020 microprocessor inthe same VME crate. Additionally, several different typesof serial devices are connected directly to IOCs using thefront panel serial ports. In order to support thiscomplement of hardware, device support routines must be
incorporated into EPICS, and, in some cases, custom
VxWorks kernel must be built.
EPICS provides the basis for our control system in theform of the core code which executes and manages thedevice control applications which run on the IOCs, thecommunication layer software for name based I/O known
as Channel Access, and the generic tools for graphical userinterfaces, alarms, datalogging and backup and restore. For
device control we have developed, using EPICS, 120different applications. In many cases, similar hardware
configurations are repeated throughout the machine, andthe EPICS device control applications are replicated and3 Machine availability
Jefferson Lab’s goal for total machine availability for
Fiscal Year 1997 (FY97) was 70% uptime. Time iscounted as uptime when the accelerator is delivering beam
on a target in at least one of the three experimental halls. Inorder to support this goal, the Accelerator Control System
had a goal of 98% uptime during scheduled machine
operations. Unfortunately, and despite much hard work,
we fell short of both goals. The actual machine availability
for the year was 58.6% during 41.8 weeks of scheduled
operations. Control system availability, at 94.3%
contributed 402 hours of downtime, almost 262 hours over
our goal. Clearly, this excessive downtime must be
eliminated in order to support overall machine availability
for physics experiments. Figure 1 shows control system
downtime by month. The month of January is absent fromthe graph since this was scheduled downtime for machine
maintenance and upgrades.
In order to improve the availability of the control system,

we track causes of downtime, and focus our efforts onsolving the problems which cause the most downtime.
Downtime is tracked by an automated program called
Control System Downtime by Month

84:00
72:00
60:00
48:00
36:00
24:00
12:00

SepAugJul

JunAprMayMarFeb

DecOct0:00
Nov

loaded on multiple IOCs. A total of 850 instances of theseapplications are loaded on the 80 control system IOCs thatrun the accelerator.
On the Unix side, in addition to the standard EPICS tools
mentioned above, a number of additional system and highlevel applications are utilized for machine operations.
Some of these applications also use commercial software
packages such as Matlab and ObjectStore. Other
applications make use of freely distributed software suchas the GNU tools, Tcl/Tk and the World Wide Web. Inaddition to the complexity inherent in this system of somany integrated pieces, is the problem of how to maintainstable control system operation while making frequent
upgrades to the existing hardware and software andintegrating completely new systems.
The standard operating schedule calls for machine
operations 24 hours a day, 7 days a week with one 8 hour
shift for maintenance every two weeks. Within this single
shift, all hardware and software upgrades must be installed
and tested. Additionally, we must provide the ability toreturn the control system to the previous operational state
if testing during an upgrade reveals a problem.
The effect of a program error on the Unix side is usuallythat a single program or tool is unavailable until repair.
Frequently, operations can continue in these cases. On theIOC side, the effect of an error or a hardware failure is far
more damaging as the entire IOC is general rendered
unavailable to operations until a repair is made. It is almost
always the case that this type of problem stops machine
operations and counts as accelerator downtime. For thisreason, we have concentrated our initial efforts to provide
software versioning and the ability to quickly roll backapplications on the IOC software. This system is now inplace and works very well. There have been many times
when the rollback mechanism has been quickly andsuccessfully invoked to return an IOC to operations when
new software has introduced a bug. Now that this system isworking well, we have turned our efforts in this area to theUnix systems.

MonthFigure 1. Control System Downtime by Month
Downtime Logger and by operator input to this program.
Using this information, we are able to allocate resources toreduce the most costly sources of downtime, andimplement tools and procedures to address underlying
causes of greater downtime such as troubleshooting timeand change management. The data, which consists of 210
control system downtime entries this fiscal year, has been
analyzed to determine the failures that were the single
longest causes of downtime, and also the repeating causes
which take only a short time to recover from, but which
occur so often as to have a large cumulative effect onoverall downtime. For FY97, the top five longest control
system downtimes account for 32% of the total 402 hours
of control system downtime and are shown in Table 1.
Table 1. Top Five Downtime Events
Downtime Cause
CAMAC Power Supply
Fiber Optic U-Port
CAMAC Power Supply &L2 Controller
Disk Failure
Network Hub Failure
Total

Time (Hours:Minutes)
39:24
25:34
22:14
21:20
21:01
129:33

The first three entries in this table show many hours ofdowntime for simple replacements of standard hardware
parts kept on hand. The time incurred here is the total time
to diagnose and repair the problem and the time to restore
the machine to the previous running condition. In two ofthese cases, time to determine the failed component waslong. Each of these events also resulted in a crash of theCentral Helium Liquefier (CHL) Plant due to loss ofcontrol, which added additional recovery time. In the caseof the network failure, additional time was needed toobtain replacement parts. The disk failure problem waslengthy to repair because the failed disk was a root volume
without a spare copy built. This problem was complicatedby failures of the backup system in the preceding days
making file recovery very difficult.
Grouping downtime events into categories, also helps toidentify costly problems that are not so obvious. The data
for the top five categories of downtime, as shown in Table
2, accounts for almost 77% of all control system
downtime.
This data also reveals that the most significant category ofcontrol system downtime, IOC problems does not evenshow up at all in the top five most time consuming failures.
Table 2. Top Five Downtime Causes By Category
Downtime Cause
IOCNetwork
CAMAC Serial Controller
CAMAC Power Supply
BPM Mux VME CardTotal

Time (Hours:Minutes)
110:28
66:28
61:49
48:30
21:37
308:52This is because IOC failures do not usually take such along time to recover from, but at times occur quite
frequently. Of the 210 control system downtime entries
logged in FY97, 113 were classified as IOC problems. One
reason for such a large number here is that IOCs are themost visible part of the control system, and often, problems
that really originate elsewhere are logged as IOC problems.
Failures of the network, VME modules or VME crate
power supplies are often first thought to be IOC problems.
While we try to update or correct downtime entries when adifferent problem cause is found, this is difficult sincemachine operations are often restored long before the root
cause of a problem is discovered.
Despite such reporting errors, IOCs still constitute amajor source of control system downtime. Unlike the Unix
programs, the effect of an error in a program running on an
IOC is generally that the entire IOC and all applications
running there fail to function when the bug is reached. We
have discovered and fixed two such bugs during thisreporting period which account for well over half of thereported IOC problems. These types of problems are
generally difficult to isolate and debug due to thenumerous applications running on each IOC and various
system interactions.
The network also proved to be a significant source ofcontrol system downtime. This was due to a combinationof actually hardware failures and configuration problems.
These failures have proven time consuming totroubleshoot, and for that reason, we have added additional
network monitoring tools and are training more people on
this system.
The fact that crate power supplies contribute almost 50
hours of downtime can be traced, at least in part to the ageof this equipment. Most of the CAMAC crates installed in
the machine are now approaching ten years old. We are
now systematically replacing and monitoring these power
supplies in an attempt to reduce operational failures. TheCAMAC Serial Crate Controllers have no corresponding
reason to account for over 60 hours of downtime as mostof these cards are less than four years old. We have foundthat controllers supplied by some manufacturers aresignificantly less reliable than others, and are replacing as
many as possible with the better brand.

The VME Mux Controller cards were found to havepoorly soldered connections, and since this problem has
been fixed, this cause of downtime has been eliminated.
4 Downtime reduction
In order to reduce downtime attributable to the control
system, we have examined the causes and identified thebiggest time sinks, underlying difficulties and trends. We
have established the following steps in an ongoing effort toimprove troubleshooting and communication and minimize
downtime.
•••

•

•••

Work towards a uniform installed base of hardware
and software wherever possible. This policy helps by
reducing the potential sources of software failures bystandardizing, and reduces the number of different
types of hardware that must be understood andmaintained for spares.
Minimize the number of changes made at one time.
This policy speeds the troubleshooting process
because fewer changes means fewer items to workthrough to determine the cause of problems in apreviously operational system.
Phase in changes after testing in a limited area. This
policy reduces recovery time in cases where newsoftware or hardware must be removed from thesystem to restore operability. It is significantly fasterto rollback software on 3 IOCs than 80.
Document installation, testing and recovery
procedures for new features. This policy speeds repair
time by providing a written procedure for how toremove a new feature. This usually enables any onsite or on-call software personnel to restore a systemrather than requiring a particular individual bereached. This policy is currently only utilized forsoftware, but we believe there is a corresponding
benefit for hardware as the effect is to ensure thework and testing is well thought out and documented,also improving communication between groups.
Providing these testplans alone has reduced theinstances of software rollbacks because quality has
improved through this preplanning effort.
Schedule Control System testing time along with
Machine Recovery. Experience has shown us that theearlier a problem is discovered after machine
maintenance, the lower the operational impact. By
extensively testing control system upgrades duringmachine recovery time, we have often avoided
machine downtime since problems could be resolved
before scheduled beam delivery. One form this policy
has taken is the use of standardized checks of all IOCs
and CAMAC modules during machine lockup. This
has allowed problematic IOCs to be restored before
instead of during beam operations.
Maintain critical spares. Having on hand tested spareparts, organized in an accessible location speeds
repair time once a hardware problem is identified.
Preventive Maintenance. Power supplies on VME andCAMAC crates are now routinely checked and•••

repaired as needed. Filters are periodically cleaned
and fans replaced in an effort to prevent operationalfailures.
Provide on-site support during machine recovery. The
groups responsible for control system hardware andsoftware now provide on-site support during machinerecovery after a maintenance period. This eliminates
the time required to call in a person fortroubleshooting and repairs.
Improve and add tools to assist with troubleshooting.
Many problems we have encountered this year have
proven difficult to troubleshoot. We have developed
several new software tools to assist in this area. One
program monitors the response of CAMAC modules
and reports errors. This has proved a valuable
diagnostic tool for failures of individual CAMACmodules, and allows bad modules to be quickly
identified and swapped, virtually eliminating
troubleshooting time. We plan to further enhance thisprogram to detect problems with the CAMAC serial
highway. A second program checks indicators of IOC
problems, such as suspended tasks, corrupted stacks
and low memory, and reports any problem areas. This
allows for a quick check of all IOCs and is often used
as a first step to troubleshoot individual IOC
problems. We also plan to add more extensive checks
to this program. A third program, called IOC CoreDump, is used to capture valuable information about
IOC problems. The program stores status informationfrom the IOC and the contents of the IOC memory.
Once this program has completed running, the IOC
can be rebooted and returned to operation, and thedata can be analyzed off-line. This method was used
to find and fix the two software bugs that wereresponsible for a great deal of IOC downtime.
Improve software detection and reporting of failures.
In the cases of the frequently failing BPM MuxControllers, the software was modified to identify thebad controller, raise an operator alarm and continue
running, avoiding the bad card. This was a huge
improvement over the original program behavior•

which disabled the entire IOC. We also added alarms
to warn operators of problems such as low memory on
an IOC to give time for a controlled reboot instead of
surprise crash.
Identify and focus effort on fixing the most time
consuming problems. This approach has already been
discussed in the data analysis. This is a necessity
since available resources are not enough to address
every problem at this time.

5 Conclusions
Control system downtime has exceed our target goal byalmost a factor of three this year. We have also observedthat problems temporarily increase after machine
downtimes due to changes and upgrades, and have focused
additional effort during these times. Since these steps werefirst outlined in April 1997, the trend for control system
downtime has improved. Dividing the year into two six
month periods, before and after these steps, shows that
downtime during the second period is significantly lower.
October 1996 - March 1997 accounted for 236 hours ofdowntime, and April 1997 - September 1997 had 166
hours. The first six month period really contains only fivemonths January was a scheduled maintenance period. Eventhe smaller number of hours for the second half of the year
still exceeds our goal of 2% control system downtime, andwe will continue our efforts to make improvements.
Acknowledgment
Thanks to Tom Hassler who provided data about overallmachine availability and Steve Suhring who provided
detailed data for this paper.
References[1] K. S. White, H. Shoaee, W. A. Watson, M. Wise,"The Migration of the CEBAF Accelerator ControlSystem from TACL to EPICS", CEBAF Control
System Review (1994).