OPERATIONAL EXPERIENCE WITH THE CEBAF CONTROL SYSTEM*K. White, D. Kehne, Continuous Electron Beam Accelerator Facility, Newport News, VA 23606 USAAbstract
This paper describes our operational experience with the CEBAF control system. CEBAF is operated using a variety of software packages and custom applications. In the spirit of collaboration and software sharing, the current system utilizes codedeveloped at four major laboratories as well as commercial packages and custom applications. The basis of the CEBAF control system is EPICS (Experimental Physics and Industrial Control System), originally developed at LANL and ANL. The
CEBAF accelerator is operated primarily using the common EPICS tools such as the archiver, alarm handler and display manager. In order to supplement the features of EPICS, custom applications have been developed and incorporated into the control
system using C, C++ and TCL. Additionally, the CEBAF helium plant is controlled using the CEBAF TACL system which
requires communications between the two control systems. CATER, a problem tracking program developed at SLAC, has also
been integrated into the operational software. Current plans call for the integration of an on-line logbook developed at APS andthe internal development of a down-time logger program. We discuss how these various tools and applications are used operationally, the advantages and disadvantages of the systems and challenges related to integrating this diverse array of software.
Introduction
CEBAF is a 4 GeV CW, high luminosity electron accelerator in Newport News, Virginia. The accelerator consists of two 0.4
GeV superconducting RF linacs connected by two 180o arcs;. Each linac consists of 20 liquid helium vessels (cryomodules),each containing 8 cavities. The 45 MeV injector contains an additional 18 superconducting cavities. The beam is recirculated
through the linacs up to 5 times yielding an energy of 4 GeV. After the 5th pass, the beam will be split and delivered to 3 halls
simultaneously. The 3 experimental halls house a variety of complex detectors. The control systems for both the accelerator
and the experimental facilities have recently been migrated from in-house developed packages to systems based upon EPICS(Experimental Physics and Industrial Control System). This paper describes the operational experience with the new control
system and the additional tools which have been added to supplement the basic EPICS package.

BackgroundIn the summer of 1993, in the midst of accelerator commissioning, it was decided to replace the CEBAF in-house control system TACL (Thaumaturgic Automated Control Logic) with EPICS, the product of a mult- lab collaboration. The integration ofEPICS was accomplished using a phased approach which allowed TACL and EPICS to coexist in the same operational environment.[1] This allowed commissioning to continue with a minimum of interruption from control system changes. Due to theuntimely nature of this change, many of the operational aspects of the control system were not fully developed in time forcomplete machine operation. While a number of EPICS tools have been successfully used, both for commissioning and operations, other programs have been needed to supplement the EPICS capabilities. Additional programs have been developed bothin-house by programmers, operators and scientists and adapted from other laboratories.

I.

THE EPICS TOOLSEPICS includes a number of utility programs which run under Unix and communicate with IOCs (Input-Output Controller)
over Ethernet using channel access software. The following describes each tool, how CEBAF has used them and whereimprovements would improve machine operability.
Graphical User Interface
The EPICS tool MEDM (Motif-based Editor and Display Manager) is used in producing and animating graphical user interfaces. CEBAF has developed over 800 operational display pages using this tool. These pages form the biggest component ofCEBAF’s operational user interface. MEDM provides the ability to quickly develop and modify pages to meet changing
requirements. The editor tool has a menu driven, point and click, WYSIWYG type of interface which requires no programming, so operators and hardware support groups are able to develop or customize display pages for their own individual needs.
MEDM is cumbersome to use when developing screens with a large number of display elements, a problem that has beenaddressed by an engineer who has developed a library of subroutines which allow screens to be generated programmatically.[2] CEBAF has added several features to MEDM, such as bit displays, remapping of visible windows and zero centering
bar graphs to customize this tool for our needs. EPICS provides two independent tools and work has begun at LANL to merge*Supported by U.S. DOE Contract DE-AC05-84-ER40150

these to produce a tool with the features of both. CEBAF will collaborate in this effort.
Data Archival and Archive Data Viewing
The EPICS data archiver tool, AR, is used at CEBAF to log values continuously, or to log values upon some condition. The
EPICS archive viewer tool, ARR provides some basic display capabilities for a very limited amount of data. Both the archiver
and archive viewer in their current form need a great deal of improvement to be useful and reliable for operations. The callable
interface to the archived data needs improvement to make the data more accessible to other analysis programs. The EPICS
archive viewer does not have enough capacity to be useful for operations. CEBAF has already written an X-based archive
viewer, XARR, that provides basic viewing capabilities for EPICS archived data. This tool needs to be improved and morefeatures added. The EPICS collaboration plans to replace AR and ARR, and CEBAF will participate in this effort. Despite
problems with these tools, about 42 megabytes of data are logged each day by operations. Some of this data is used to characterize the machine and some is used by support groups to analyze problems and study their systems.
Alarms
ALH, the EPICS Alarm Handler, is used to notify the operations staff of current or potential problems with machine hardware
and software. The magnet system, with over 2000 elements, is the largest system monitored by ALH. For each magnet, twodifferent parameters are monitored. Other systems incorporated into the alarm handler include Viewers, Vacuum and Valves,and Controls. The alarm handler also includes the ability to log alarm data and this information is used by support groups todetermine which components of the system may potentially fail, or are inherently unreliable. The most frequently used alarm
handler is the magnet page. This page allows rapid detection of magnet errors that can cause critical, though subtle, changes inthe electron beam optics. These errors can be disguised by the orbit feedback loops that are used to correct the orbit througheach arc.
Save and Restore
BURT, the EPICS Backup and Restore Tool is used to save and restore operational machine parameters. In general, this tool
has worked well and proven to be reliable at saving and restoring data. Many thousands of data files have been saved for various parts of the accelerator and to reflect different configurations for operations. While the tool itself works well, it lacks any
useful user interface. In order to solve this problem, operations personnel developed an interface to BURT using TCL. This
interface makes it trivial for an operator to save or restore predefined sets of parameters from any combination of geographicalregions of the accelerator. It also allows saved files to be compared for differences.

II.

THE TACL TO EPICS INTERFACE

In order to facilitate a smooth transition to EPICS, an interface which allowed TACL displays to access EPICS data was developed. This allowed the new systems to be tested using existing interfaces and allowed EPICS databases to be installed andoperational before time-consuming operator interfaces were completed. The CEBAF cryogenic system also remains underTACL control, so this interface allows communication between the two control systems. While this approach allowed asmooth transition and installation of new software during machine commissioning, it added to the work of the operations staff.
Firstly, having two control systems meant training all new operators to use both systems. Furthermore, the two systems run on
two physically distinct consoles meaning the operator has to use two different keyboards, screens and mice. As more of thecontrol system has been converted to EPICS, expertise with the TACL systems is decreasing, making troubleshooting difficultand more time consuming.

III.

SOFTWARE DEVELOPMENT BY OPERATIONS AND ACCELERATOR PHYSICS STAFF

Due to the overwhelming amount of work involved in changing control systems, many of the high-level applications software
programs are prototypes developed by Operations and Accelerator Physics personnel. Otherwise, these applications could nothave been made available in time for commissioning. One important result was a closer working relationship between the software and accelerator operations groups. This situation brought on the additional advantage of allowing rapid feedback on algorithm errors found during use of the application. In general, an application was developed when it became necessary or would
greatly enhance commissioning. The major applications developed in this way are for save/restore (BURT GUI), orbit andenergy feedback locks, automatic and noninvasive cresting of the linac cavity phases (KREST program), Linac Energy Management (LEM) for setting energy and optics in the linacs, automatic orbit steering of the linac and arcs, comment data basefor RF cavities and beam position monitors and on-line procedures. Each of these is discussed in more detail in the followingsections.

Display Generation Library
As discussed in the section on EPICS Tools, the use of MEDM is inefficient for building complex screens with a large numberof signals. In order to address this problem, a library of C functions was developed by an Operations crew chief. These functions can be used to generate display pages. This allows the quick development of complex screens and allows similar screens
to be developed quickly, simply by loading a new set of signal names and executing the program again. Not only have thesefunctions saved many hours in generating display pages, they have also created more uniform and effective displays than
could have been created by hand. In general, the pages created by using these programs contain far more information content
than comparable hand drawn displays.
Slow Feedback Locks
The orbit and energy feedback locks were developed in TCL by a single member of the Accelerator Physics Group[3]. Thelocks were found to be necessary when slow drifts of the RF over the span of minutes to hours made the setup of the optics
extremely tedious. Status of the RF had to be monitored and adjusted every few hours. The application had immediate impact
on improving the efficiency of commissioning. Launch angles out of the injector, linacs and arcs can be reproduced. This
allows the energy to be set and accurately locked to all beam position monitors in the arcs. One major drawback is that it can
only be effectively maintained by the person who wrote it. Moreover, since it was developed rapidly, the program does nothave mature error handling, thus increasing the number of faults the program experiences.
On-line Maintenance Comments
CEBAF has a total of 338 superconducting cavities, only two of which are individually critical to beam operation. As cavities
fail, they are taken off-line and then repaired during scheduled machine maintenance periods. Though a summary of the statusof every cavity in the accelerator is shown on an MEDM screen, the most effective method to access a more detailed history ofindividual cavities was to establish a database that could be easily modified by the person shutting down or repairing the given
cavity. In order to easily access and update the database, a TCL interface was written using key buttons and comment fields. Inaddition, the name of the person entering the comment as well as the date are automatically recorded. Installation of this database has proven to be an effective method of communicating cavity status and problems to the RF maintenance group, as wellas to operations. It has also been implemented to track detailed status for beam position monitors.
Automated Linac Phasing
The KREST application arose when drifting phases in the RF, due to diurnal temperature changes, caused unacceptable energy
spread in the beam.[4] Depending on the request file, KREST will change the phase of individual cavities, cryomodules or anentire linac. After changing the phase, KREST monitors the cavity gradients used by energy lock to see how the cavities
respond. By changing the phase, the KREST program can determine where the peak of the RF phase is located. Since eachcavity phase need only be changed by small amounts to achieve the desired precision, the total energy in the accelerator is notaffected significantly. Therefore, assuming the energy locks are running, KREST can usually be run in the background duringcommissioning. If a cavity gradient becomes unstable or beam energy is deliberately changed while KREST is running, a cavity can be left substantially off-crest. Often this condition is not discovered unless one specifically pulls up the KREST historyfile. KREST takes approximately 2-3 hours to crest all cavities in a linac.
Save/Restore Interface
In order to provide a usable interface to the EPICS save and restore program, a physicist develop the BURT Graphical User
Interface (BURT GUI).[5] This Tcl/Tk application displays a list of files available to restore, along with the date each file wasupdated and the name of the operator who saved the file. A comments field is also displayed. This interface also allows two
BURT files to be compared for differences. This has proven to be a valuable diagnostic tool.
Linac Energy Management
During early commissioning, it was soon discovered that cavities were being lost at a rate of 3-6 per day. When a cavity is lost,the energy must be compensated for elsewhere in the linac and the optics, set for 120o phase advance per period, would usually
have to be reset. Resetting the linac optics manually, after much practice, could be done in one half hour. The Linac Energy
Management (LEM) program was written to automatically redistribute the gradients and recalculate and load the optics using
the optics code DIMAD. As with the slow feedback locks, the program is maintained by one individual and is not robustlycoded.
Beam Steering
The AUTOSTEER program serves several purposes. It is used primarily to minimize the orbit through certain sections of themachine, such as an arc or linac. In the linac, autosteer simply flattens the orbit of the first pass. In the arcs, energy lock notonly flattens the orbit but can also be used to measure changes in beam energy to 10-4. Absolute energy measurement accuracyis ~10-3. Effects of the earth’s field can be included in the calculation of the orbit and energy but this causes an error in themeasurement of the absolute energy.

On-line Procedures
Instead of the traditional method of having a notebook of procedures on paper, the CEBAF operational procedures are written
using FrameMaker and utilize hypertext links for document organization. This allows the procedures to be displayed on the Xterminal alongside other control system displays. The operator can always find the procedures and modified versions can be
stored on-line as a procedure is updated. Operational experience has shown that it is more straightforward to have hard copiesof procedures to make comments on rather than a framemaker file on screen. Therefore, there is also the option to simply printout the procedure and marked up hardcopies are then directed to the authors.

IV.

APPLICATIONS ADAPTED FROM OTHER LABS

In addition to joining the EPICS collaboration, CEBAF strives to take advantage of common needs and adapt programs fromother labs to reduce software development time and expenses. We have successfully integrated a trouble reporting programand an optics diagnostic tool from SLAC (Stanford Linear Accelerator). We have plans to adopt the electronic logbook used at
the Advanced Photon Source (APS).

Trouble Reporting
Once commissioning was well underway, the Hardware Reliability Team recognized a need for a database-driven problem
tracking program. With a small amount of software work (as compared to a full development) to add custom menu items andhelp, a program written at SLAC was adapted. The CATER (Computer Aided Trouble and Error Reporting) program has now
been in use for two years at CEBAF. During this time, nearly 4000 trouble reports have been entered and routed to the appropriate groups for action. This program has provided a consistent method for communication between operations and support
groups and allows on-demand tracking of problems. Since the information is stored in a database, it is available for trending
and failure analysis. The main difficulty with this system is the lack of a modern user interface and analysis tools, thus making
it difficult to perform some functions. In addition, CATER is a VMS-based application, making any future integration with
UNIX-based control system tools difficult and prohibitively expensive.
Optics Diagnostics Tool
For on-line analysis of beam orbit deviations, the RESOLVE program developed at SLAC has been imported to CEBAF. By
experimentally producing orbit and energy changes at certain points in the accelerator, deviations of the machine orbit compared with the orbit analytically calculated by RESOLVE can quickly indicate a faulty element. Like CATER, the version ofRESOLVE currently used at CEBAF operates on a VMS machine. Therefore, data must be transferred to that computer, substantially slowing the analysis process. In addition, the current program configuration allows only approximately half a pass ofoptics to be analyzed at a given time. Significant amounts of data manipulation must be done to observe orbit changes throughmultiple passes, thus making the analysis turnaround too lengthy. Both of these problems are being addressed.
Electronic Logbook
CEBAF is currently planning to adopt the an on-line logbook similar to the e-log used by the APS at Argonne. The system uses
FrameMaker and World Wide Web to enter and display control room logbook information. The use of these tools coupled witha scanner will allow paper and computer documents as well as on screen graphics to be quickly incorporated into the logbook.
The hypertext capability will allow better information organization. Other advantages of having the logbook on-line are theability to use computer search functions and the ability to display the information to multiple users simultaneously.

V.

CONCLUSION

In addition to the EPICS toolkit developed by a collaboration of laboratories, CEBAF has used software from a variety ofsources to commission and operate the accelerator. The use of X-Windows allows many different applications to be used in acommon environment. Software has been written by programmers, operators, physicists, and students. Scientists and support
groups work closely with software developers to develop requirements for new projects. Operations personnel have taken an
active role in developing protoypes of programs immediately needed for operational use. This method has allowed operations
and commissioning to proceed while more robust and full featured applications are developed by the Controls Software
Department. Several programs have also been adapted from other laboratories for use at CEBAF. Only by taking this teamapproach to integrate applications from many sources has the controls software reached a level sufficient for machine operations after a mid-commissioning control system change.

VI.

REFERENCES[1] Karen S. White, et. al., “The Migration of the CEBAF Accelerator Control System from TACL to EPICS”, CEBAF Control System
Review (1994).
[2] Jeff Karn, “Code Driven Construction of MEDM Screens”, CEBAF-TN-95-038 (1995).
[3] Johannes Van Zeijts, “Rapid Application Development Using the Tcl/Tk Language”, (PAC 1995).
[4] M. Tiefenback, K. Brown, CEBAF Internal Documentation.
[5] Bruce Dunham, “Control System Data from an Operational Point of View”, (Proceeding of this conference).