Proton Storage Ring Control System Upgrade -- Interim Status†
Michael A. Oothoudt, Eric A. Bjorklund, Stanley K. Brown, Norman T. Callaway, Gary P. Carr,
L. Robert Dalesio, Dolores B. Duran, John A. Faucett, David P. Gurd, Matthew W. Hardy,
Margye P. Harrington, Jeffrey O. Hill, Debora M. Kerstiens, Andrew J. Kozubal,
Franklin J. Naivar, Matthew C. Needes, David J. Ostrem, Jerry M. Potter, Bobby A. Quintana,
Stuart C. Schaller, Fred E. Shelley Jr., Matthew W. Stettler, Michael E. Thuot, George D. Vaughn,
David S. Warren, Robert E. Weiss, Rozelle M. Wright, Martha V. Zumbro
Los Alamos Neutron Science Center, Los Alamos National Laboratory, Los Alamos, New Mexico 87545

The Los Alamos Neutron Science Center (LANSCE) is in the process of a significant upgrade to its facilities.
As part of this upgrade, the control system for the Proton Storage Ring (PSR) is being upgraded to an EPICS-based[1]system following the standard control system model. Goals of the overall upgrade required implementation of the new
system with special attention to maintaining the functionality and high availability of the old system. The first stage ofthe control system upgrade has replaced about 75% of the old system and has been in high availability use for over twomonths. The old and new systems are briefly described and the strategies used to meet the upgrade goals are
discussed.

1. INTRODUCTION
The LANSCE facility (formerly known as LAMPF, the Los Alamos Meson Physics Facility) provides 800
MeV proton beams for several experimental areas. A principal facility is the Manuel Lujan Neutron Scattering Center(MLNSC), where a high peak current proton beam is scattered off a tungsten target to produce intense neutron beams
for materials science studies. The high peak-current proton beam is produced by accumulating beam from the
LANSCE linac in the Proton Storage Ring (PSR) and extracting the stored beam in a single 270 ns turn.
In 1994 an upgrade of the PSR was begun to improve beam delivery for MLNSC. The goals of the upgrade
relevant to the control system are• routine, sustained 8 months/year operation of LANSCE,• beam availability >80% by 1996; >85% by 1998,• less than 5% downtime from intervals >8 hours,• reduced beam delivery operating costs, and• 100 µA routine operating current at 20 Hz by 1998
An upgrade of the PSR control system was approved as part of the overall upgrade and work started in mid
1994. Budget, available manpower and the LANSCE operating schedule precluded performing the entire control
system upgrade before beam tuneup began for the 1995 production period. An interim architecture was installed thatsimultaneously supported all old and new software and hardware in the control system. The hybrid system was
successfully fielded in June 1995 for testing, equipment checkout and beam tuning. The new system has been inhighly reliable beam production for over two months.

2. THE ORIGINAL PROTON STORAGE RING CONTROL SYSTEM
The architecture of the original PSR control system[2] is shown in Figure 1. The system used a centralized
database on a VAX computer running VMS. There were approximately 3500 hardware channels and 400 software
channels in the system when the upgrade began. Approximately once every 2/3 of a second the VAX wrote allcommand channels to the remote computers and read all output data from the remote computers over a CAMAC serial
highway. The PDP-11 remote computers commanded and read CAMAC modules in CAMAC crates on their local
serial CAMAC highways. In addition the VAX directly addressed CAMAC crates on the VAX serial highway; thesecrates interfaced to knobs, operator displays, the ring buncher RF wave-form generator and the ring Beam Position
Monitor (BPM) system. Other data were acquired and commanded over Ethernet from the Linac Control System(LCS). The LCS is an independent system using a demand-based architecture[3].
† Work supported by the U.S. Department of Energy.

The operator interface to the control system was through dumb terminals, Elographic touch panels on
Lexidata display screens and knob panels and these devices resided on the VAX Unibus or the VAX CAMAC serial
highway. In addition ,operators had access to LCS facilities through peripherals from that system. Over the lifetimeof the system the central computer was upgraded from a VAX-11/750 to a VAX 4000/300 and the remote computerswere upgraded from LSI-11/2 in-crate computers to standalone PDP-11/73s.

LCS
Ethernet
knobs

Display

BPM

RB RF

PSR VAX
PDP

PDP

PDP

CC

CC

CC

CC

CC

CC

CC

PDP

CC

CC

CC

CC

CC

CC

CC

CC

CC

CC

Figure 1 - Architecture of the PSR control system before upgrade. "LCS" is the Linac Control System. The ellipses
are CAMAC serial highways. "CC" is a CAMAC crate. "BPM" is the interface to the Ring Beam Position Monitor
System. "RB RF" is the ring buncher RF waveform generator.
An upgrade of the control system was desired for several reasons. Applications were, in general, developedin Fortran by the programming staff and even they required long, expensive periods of time to field new applications.
The high cost and long turn-around time of new applications were considered major flaws of the system. Furthermore,while the original control system had good availability (99.7%) in the recent past, signs of end-of-life failures werebeginning to appear, including multi-day down-times that fortunately occurred when beam was not scheduled fordelivery. Because much of the hardware was 10 or more years old, manufacturer support was limited and in somecases was no longer available. Maintenance costs were rising because of lack of manufacturer support and dwindling
local expertise. Failures of the long CAMAC serial highway were difficult to troubleshoot. Lexidata displays showedintermittent failures that were never adequately explained. Hardware features (such as low maximum program
memory in the PDP-11s) limited ability to expand the system. Finally, the periodic migration of data limited the
responsiveness of the system; e.g. the minimum possible response time for knobbing a channel was 1.33 seconds.

3. THE NEW PSR CONTROL SYSTEM
The new system is based on EPICS[1] software and a distributed architecture. Figure 2 shows thearchitecture of the new control system as implemented for 1995.
3.1 Hardware
For budgetary reasons the individual serial CAMAC loops and crates have been retained. However, it waspossible to reconfigure some of the serial loops to reduce their lengths and to remove crates of harp electronics nolonger in use; this has reduced the incidence of serial highway problems and made trouble shooting problems easier.
Several unreliable stepper motor modules were also replaced with Allen Bradley (AB) DACs; this change is still underdiscussion, since some system experts believe the improved device control does not adequately compensate forrunning magnets to zero current when power to the DACs is lost.

Remote computers have been replaced with Motorola 68040 monoboard I/O Control (IOC) computers in
VME crates. The IOCs run local databases that read and control the modules on their local CAMAC serial highways
and Allen Bradley crates. Most data in this system is polled by the IOCs at 1 to 5 Hz; this rate may be increased to
improve responsiveness of the system when the system is complete enough for optimization. Programs on other IOCsor Operator Interface (OPI) computers obtain data through EPICS Channel Access by posting interest in particular data
fields; data is shipped by an IOC to interested programs over Ethernet when changes occur outside predefined
deadbands. Building the IOC databases to provide the functionality of the old control system was the major part of the
upgrade effort.

EPICS
OPI

LCS

EPICS
OPI

AB

EPICS
IOC

Ext
BPM

EPICS
IOC

Ethernet

PSR VAX

EPICS
IOC

EPICS
IOC

Display

knobs

CC

CC

CC

BPM

RB RF

CC

CC

CC

PDP

PDP

PDP

PDP

CC

EPICS IOC
CC

CC

CC

CC

CC

CC

CC

CC

CC

CC

Figure 2 - Architecture of new PSR control system used in 1995. "LCS" is the Linac Control System. The ellipses are
CAMAC serial highways. "CC" is a CAMAC crate. "RB RF" is the ring buncher RF waveform generator. "AB" is an
Allen Bradley electronics crate. "BPM" is the interface to the Ring Beam Position Monitor System. "Ext BPM" is theExtraction Line BPM system. PDP-11s are present for the fallback strategy.
3.2 Software
The Operator Interface (OPI) is provided by Sun Sparc 5 workstations communicating with the IOCs over
Ethernet. Two OPIs are provided in the control room PSR console with two 20 inch monitors and a mouse on each.
Screens to replace old PSR applications were mostly developed with the EPICS edd screen editor and displayed withthe dm display manager. Four screens requiring data and file manipulation not provided by edd/dm were built with the
tcl/tk package[4]. All but five of the old PSR graphical applications were converted for 1995 operations.
Three new applications were installed as part of the upgrade plan: a display of the state of the Radiation
Security System logic, a display of beam spill monitors in the PSR and a display of extraction line BPMs. Design andimplementation of the first two new displays took four man-weeks (compared to an estimate of 52 man-weeks in theold PSR system.)
3.3 Interchanging Data Between Systems
Many of the OPI screens require data from both the LCS and the old PSR systems. Gateway software[5]provides EPICS Channel Access to this data from the OPIs. Because of implementation details, the gateways provide
fresh data at a 0.1-1 Hz update rate rather than on-change. This update rate has been singled out by users of the system
as being too slow.
Several applications in the old PSR system were not converted to EPICS for 1995 production due tobudgetary and manpower constraints. These applications and many LCS utilities require access to data that is now in
the EPICS IOCs. Data migrator software[5] was created to periodically transfer a fixed list of data channels between
the EPICS IOC databases and the PSR/LCS databases. A side effect of implementing the migrators was that
implementing knobs in EPICS became unnecessary for 1995; the existing LCS knob system could control EPICS

devices through the migrator path. Furthermore, the problem of time correlating data stored to disk by EPICS and
LCS was avoided, since all data could be recorded by LCS. The added complexity of the migrator software did
increase the trouble shooting problems. However, deferring these two jobs until future years helped to meet thedelivery schedule for 1995.

4. PRESERVING HIGH AVAILABILITY WHILE UPGRADING A PRODUCTION SYSTEM
A principle goal of the overall PSR Upgrade was to improve availability of the entire PSR beam delivery
system. The PSR control system ran at an availability of 99.7% during the most recent previous beam production
period. The upgraded control system was expected to run with at least this availability in 1995, while maintaining allfunctionality of the old system. Higher availability in future years was also expected. These challenging goals for anew, hybrid system were attacked by a series of efforts:• Options Review: As part of the planning process in early 1994, a thorough review[6] was made of previous
proposals for PSR control system upgrades and new possibilities were identified. A total of seven options wereidentified, including "do nothing." Most of the options were expected to become as obsolete as the current system
within a few years. The EPICS tool kit was selected as the basis on which to build the upgrade.
• External Review: After an initial plan for the upgrade was constructed, an external review was held. Control
system experts from CEBAF, DESY and SSCL reviewed the proposed plan and could "identify no showstoppers,"
but made several suggestions that were folded into the planning.
• Fallback: The requirement for high availability of the new system left little time for debugging the system once
beam delivery began. It was conceivable that running with beam could show flaws, in spite of testing without beam,that could prevent the initial tuning of PSR. Therefore a "fallback" strategy was developed such that the control
system could revert to the old PSR control system if necessary. As shown in Figure 2, PDP-11s and old display
systems were maintained in the system for 1995 operation. Hardware for the new IOCs was installed such that asingle set of cables (the local CAMAC serial highway) could be manually swapped between an IOC and PDP-11.
The migrators insured that the PSR database contained a copy of all current setpoints. The fallback system was tested
before beam delivery started and was demonstrated to resurrect the old control system in less than 20 minutes. While
the fallback system did meet the requirement for assured operation, control system developers felt it incurred extra
development costs and limited enhancements to the system.
• Collaboration: The budget for training the developers on EPICS was limited to 6 one-hour classes. To provide
further training, a collaboration was formed with local EPICS experts to do the upgrade. Approximately half of the
manpower used to construct the system came from the EPICS experts. Highly successful teams of PSR and EPICS
experts were assigned to work on projects for the upgrade. To aid this collaboration, the PSR developers moved into
the same office building as EPICS experts. The proximity of the two sets of people significantly aided cross training
and communication.
• Screen Review: The budget for training users of the new system was also minimal. Furthermore it was very
important that the system initially fielded in June of 1995 should support all functions of the old system. To addressthese two issues, a series of screen reviews was held. Screen developers met with accelerator operators, technicians,
engineers and physicists who would be using the screens. Instead of lengthy requirements and users manuals,
prototypes (sometimes very crude mockups) of the screens were reviewed. These screen walk-throughs were
exceptionally successful in catching misconceptions by control system developers and getting suggestions from
customers. Furthermore the customers were trained on using the system during the reviews.
• Customer Participation: One of the accelerator operators was detailed to work with the control system developers
for four months. He was assigned to build the screens for the major utility used to control the beam lines. His active
and detailed participation during construction of the system made a major impact on the usability of the system.
• Testing: Much of the PSR hardware was not operated during the previous 18 months. Therefore an attempt wasmade by the control system developers to check every channel in the new system before beam delivery. With thehelp of operators, an attempt was made to operate every device in the system, including tests of interlocks. A channel
list of the system was used as a check list. Problems with software, CAMAC hardware and devices were found.
Unfortunately many devices in the systems were not ready for operation, so only a subset could be tested andcorrected where necessary. Later the system owners performed another channel-by-channel checkout of the system
which uncovered further problems.
• Training: Operators were trained on using the EPICS edd/dm tools for building screens. Special captive accounts
were set up to allow using these tools without learning Unix. After beam production began, operators built a number

of very useful screens that were not in the upgrade plan. These screens made the new system significantly more
useful and aided in getting reactions from customers.
• Coordination: Up to the point of channel-by-channel checkout, coordination meetings for the upgrade were held
weekly. At this point, brief daily meetings were held to analyze problems uncovered. The problems and solutions
found were published in publicly readable forms.
• Standing Shifts: During initial tuning of the PSR, a 25% contingency was built into the schedule for working oncontrol system problems. Control system developers stood shifts in the Central Control Room 16 hours per day witha developer on-call for the remaining eight hours per day. This allowed rapid response when problems wereidentified and helped developers better to understand how the control system was being used. Customers were also
given the feeling that the system had not been "thrown over the wall."• Observation: After beam was in production, developers spent significant amounts of time in the control room
observing the use of the new control system. Assistance in use of the system was provided and problems were more
readily identified.
• Polling for Input: After the system had been in use for several weeks, reports of problems stopped coming in.
However, there were rumors of difficulties. Regular meetings with each operating crew were scheduled to poll for
any input on the system. This uncovered a significant number of "inconveniences" that were preventing efficient
operation of PSR. These polling meetings will continue throughout the operation period.
• CQI: One significant class of problems uncovered by the polling meetings was magnet instability. A Continuous
Quality Improvement team was formed with software, electronics and power supply members. This effort has
uncovered problems with software calibration, CAMAC electronics, field wiring, grounding and power supply
internal controls. Progress is being made on these problems, some of which date to design decisions made in theearly 1980s.
• Customer Critique: A meeting was held with customers to critique the new system after it was in production use.
The overall summary was that the customers "got what they needed." Two exemplary practices were identified:◊ Screen Reviews◊ Operators trained to create screens that are improving facility operation.
Areas needing improvement were also identified:◊ Final control system checkout overlapped too much with system experts' checkout of equipment. Schedule
future control system upgrade work to be completed before other people need to use it.
◊ Some screens did not get reviewed. Insure all future screens go through the review process. Review existing
screens that were not reviewed.
◊ Parameter and calibration constants need better configuration control. Make it easy for non-experts todetermine what constants are in use.
◊ Some CAMAC hardware has an unacceptably slow response time (e.g. some multichannel ADCs with 8
second cycle time). Upgrade this hardware.
◊ Only the operators were trained in screen building. Technicians, engineers and physicists also need to be
trained.
◊ More training is needed on trouble shooting the new system.
Finally, a number of changes and enhancements for the new system were identified.

5. CONCLUSIONS
Highlights of the PSR control system upgrade to date include:• After 56 days of production use, the new PSR control system has run with 99.946% availability, handsomely
meeting the requirements of the upgrade. The majority of the control system downtime was due to CAMAC
hardware problems.
• Customers said they "got what they needed" to do their work.
• Operators find the new OPI user interface preferable to the old touch panels.
• Operators are rapidly creating new screens to improve accelerator operation.
The old control system is approximately 75% replaced for 1995 operation. The cost of the work to date isapproximately seven man-years plus $88K for hardware and software purchases. Costs for other upgrade options wereestimated to be 30 to 70% higher for a similar amount of work. A rough cost breakdown shows 5% of the cost forplanning and prototyping, 47% for IOC database work, 36% for non-database work and 12% for systemcommissioning.
Work has begun to complete the upgrade by April 1996 for the 1996 equipment checkout, beam tuning andproduction. In addition to converting the ring BPM and ring buncher systems to EPICS, much of the infrastructure in
the old control system (e.g. diagnostic software) needs to be converted. Documentation, error logging, knobs andmagnet control problems need to be addressed. Speed of the gateway software needs to be improved.
A future goal is to have a single control system for the entire LANSCE facility. Outline studies have begun toestimate the scope of thework necessary to convert the Linac Control System to EPICS.

ACKNOWLEDGMENTS
The authors wish to thank Dave Barker, Johnny Tang and Mary Wise (CEBAF),Ying Wu (Duke University) and MarkRivers (University of Chicago) for providing baseline code and support for the CAMAC library, driver and device
support. Further thanks go to Johannes van Zeijts (CEBAF) for providing a version of tcl/tk with EPICS channel
access included.

REFERENCES[1] L. R. Dalesio, et al., Nucl. Instr. and Meth. A352 (1994) 179.
[2] P. Clout, et al., Nucl. Instr. and Meth. A247 (1986) 116.
[3] G. P. Carr, et al., Proceedings of the 1987 Europhysics Conference on Control Systems for Experimental Physics,
CERN Yellow Report 90-08 (1990) 107.
[4] J. K. Ousterhout, Tcl and the Tk Toolkit, Addison-Wesley Publishing Company, Reading, Mass., 1994.
[5] S. C. Schaller and M. A. Oothoudt, these Proceedings.
[6] M. A. Oothoudt, et al., Evaluation of PSR Control System Upgrades, AOT-6 Technical Report AOT-6-94-25,
Los Alamos National Laboratory, 1994.