Status Report on the TRIUMF Central Control System
M.M. Mouat, B. Davison, S.G. Kadantsev, E. Klassen, K.S. Lee, J.E. Richards, T.M. Tateyama, P.W.
Wilmshurst, P.J. Yogendran
TRIUMF 4004 Wesbrook Mall, Vancouver, B.C., Canada, V6T 2A3Abstract
This paper presents the current status of the TRIUMF
500 Mev cyclotron’ scentral control system (CCS). The
original TRIUMF CCS employed Data General’ sNova
computers and for more than 20 years Nova CPUs were at
the heart of the CCS. A modest upgrade project wascommenced in earnest in 1993 to replace these Novas andaccomplish several other goals. The status of the CCS is
described now that the Novas are removed and the othergoals have been largely met. The current hardware andsoftware configurations are discussed and the experience
gained during the upgrade is examined. Reliability,performance, error diagnosis,
and the development
environment are also described.
1 Introduction
In 1968 TRIUMF was established as a physics
laboratory jointly operated by a group of four Canadianuniversities. The primary accelerator is the 500 Mev
cyclotron which is presently capable of producing foursimultaneous beams of protons at various energies ranging
from 180 to 520 Mev and currents ranging up to 180 uA on
the high current beamline. There are both experimental
and applied programs ongoing and there is a new
radioactive beam facility (ISAC) under construction.
The central control system for TRIUMF ’ s500 Mev
cyclotron was originally installed in 1973 and since thattime the evolution of the central controls has occurred at aslow rate with some accelerated periods. The most recent
significant change started in 1993 when a project wasauthorized to replace the existing control systemcomputers. Neither the human nor financial resources werefirm but a budget and a three year schedule wereestablished. These old computers have now been replaced
and the new system is installed and running very reliably.
This upgrade was accomplished with a minimum of
disruption to cyclotron operation and no alteration to beamschedules due to control system needs. The current system
couples the newer computers and the older but successful
data acquisition hardware architecture. This is definitely
not a “ standard model” control system.
In addition to the upgraded central control system, otherdevelopments have been completed since the last time thatthe status of the TRIUMF central control system wasreported, and still other projects are underway or beingplanned. These non-upgrade projects include such items as
two somewhat isolated control systems which have been
partially integrated into the central system, the completionof controls for the Proton Therapy Treatment

Facility, and support for the new 2A extraction system andbeamline.
2 The central control system upgrade
In April 1993 a project was commenced to replace theprinciple computers and their peripherals in the 500 Mev
cyclotron’s central control system. This included replacing
6 Data General Nova computers running realtime code andanother 4 Nova and Eclipse computers used for support.
The work had to be done without affecting the cyclotron’s
operation or beam schedule and without additional staff.
Financial resources were found to cover the basic hardware
requirements.
2.1 Mandate and current status
From upgrade proposals going back more than 10 years,
a rather simple set of goals was targeted. The primary aim
was to replace the old computers. This was needed forseveral reasons. The hardware was failing reasonably
often, replacement parts were no longer available, thesoftware was written in assembler and thus hard tomaintain, the computers were slow due to their oldtechnology, and the memory address space was full so
new software could not be accommodated. It should be
noted that all things being considered, these computers
and their software ran remarkably well and only
contributed a relatively small fraction of the overall
cyclotron downtime. The larger issues were that thehardware was so old that it might break irreparably andnew software functionality could not be supported.
In addition to replacing the computers, several software
goals were identified. The three primary goals were toeliminate assembler code wherever possible (and use more
modern high level languages),
to use commercial
databases, and to employ an operating system that would
support larger address spaces. Enhanced performance andan improved user interface were anticipated.
These goals have now been achieved. At the end ofDecember 1996 the last Nova was decommissioned. The
Nova software could not really be ported because it was in
assembler so it was rewritten. The only remaining
assembler code is the VAX/VMS CAMAC driver, even
the Alpha/VMS CAMAC driver is written in C, and theVAXes will likely be replaced in due course. Oracle is
now widely used for database applications in both two andthree tier configurations. The computers in use are VAXes
and Alphas with sufficient address space and physicalmemory.
2.2 General philosophy

The central control system configuration is definitely
not a “standard model” in either hardware or software.
There is effectively just one level of computer. Theconcepts of front end computer and application computerare combined thus eliminating the need for interprocess orinterprocessor communication to access equipment. This
setup provides a very simple software and hardware
approach with good performance. Diagnosing software or
hardware problems is relatively straight forward because ofthis simplicity. Although the need for greater bandwidth is
not a requirement at present, the bandwidth can be easily
expanded in a linear fashion by adding more computers orexecutive crates.
In addition, the learning curve forunderstanding the system is relatively small, again
because of the architecture.
2.3 The hardware configurationon that segment and less than 5% on each of the othersegments.
The original data acquisition and control system is stillin use. This system uses a CAMAC executive crate which
normally supports seven CAMAC branches and allows
access by multiple computers via fast hardware arbitration.
This configuration has been replicated to support increased
requirements. Currently there are 5 executive crates with
15 serial and parallel CAMAC highways, containing bothcopper and fiber optic connections. Links to GPIB, STD,VME, and VXI are supported from the CAMAC.
CAMAC interrupts, known as LAMs, are supported andallow quick response to a variety of events. The
performance of the hardware exceeds the presentrequirements. On the newer computers, the CAMAC
standard CFSA call to access a device takes approximately

Figure 1 Hardware Configuration
The current hardware configuration is shown in figure 1.
This setup was adopted going into the upgrade project andhas proven its worth.
The computers are setup in two clusters, a production
and a development cluster. Originally there were only
VAX computers running OpenVMS but this has evolved tomixed clusters of VAXes and Alphas. Until recently, theAlphas provided the same functionality as the VAXes with
one exception, the software handling of CAMAC interrupts
had not been developed in the Alphas. This development
has recently been completed [1] and now the VAXes will
be phased out in favour of Alphas, as allowed by financialresources.
The control system equipment is bridged off from themain site FDDI backbone via a switching bridge located ina local network hub. The controls group employs 4ethernet segments. The network activity is low on all ofthe segments, with the largest component being activity to
X window terminals and running at a fairly constant 10%

25 microseconds. Device access calls at a higher level,that take longer to execute, are commonly used and thetiming of these calls depends on the complexity of theaction. To fully handle a CAMAC interrupt takes between
150 and 200 microseconds.
The Operations Group does most of its work from themain console in the control room. This console comprises
a variety of equipment including 18 X window terminals
with mouse, keyboard, and touch screen interfacing, anda variety of assignable and dedicated knobs and buttons.
In addition, there is remote console support via X window
and character cell applications.
2.4 The software configuration
The two cluster configuration has been both reliable andflexible. The clusters have similar functionality in mostrespects but are used in differnet ways. Source code forapplications is developed and maintained on the develop-

ment cluster and then when the code is ready for regularuse, only the executable image and support files aremoved to the production cluster. The use of two clusters
has advantages. There is the knowledge that theapplications can be run on development computers even if
all production computers are inoperable. Because tightcontrol is maintained over what is running on theproduction cluster,a predictable response can be
maintained and this is seen as important for operationaluse.
Predictable response is not assumed on thedevelopment cluster which allows diagnostics, beamdevelopment, and other special operational procedures tobe run without affecting regular operational aspects.
The software environment is built around VAX andAlpha OpenVMS, currently at version 6.2. The vast
majority of cyclotron information is displayed using X
window applications and the Motif window manager.
Oracle is used for a number of database applications
including the cyclotron device information,
asset
information, fault reports, message information, etc. X
window interfaces to Oracle are most common but some
web based facilities are available.
A good set of hardware and operating system
diagnostics exist. The hardware diagnostic programs aremostly developed for the CAMAC system and areextremely useful in handling faults. These programs aremaintained by the group members responsible for thehardware.
2.5 Experience gained
One perspective of the experience gained is of the issues
that went smoothly and those that were more difficult.
Most activities went well and that was a result of a numberof items. The basic architecture of TRIUMF’ sCCS was,and still is, a configuration that supports access bymultiple computers to device resources (multi sourcing).
This philosophy and implementation allows various
computers to independently access equipment knowingthat they will not interfere with the other processors. Thisrequires distributed resource lock management for those
items that need it. However, most actions are atomic anddo not need locking. TRIUMF’ sarchitecture allowed newcomputers to co-exist with the old ones and made the phaseover much easier.
The Operations and other groups were cooperative andunderstanding. This meant that time and energy were notspent in political delays. Working relatively closely with
the Operations Group meant that applications in the newsystem were generally well accepted.
Because a functioning control system existed, the basic
requirements were already defined and not much time ormoney needed to be spent on analysis. The Nova
applications were entirely coded in assembler. Withoutdocumentation, the complexity of many applications madedeciphering the code beyond difficult. For a programmer
to understand the code by simply reading it would surely
have been qualification for canonization at a later date. In
these cases, the cooperation of the Operations Group wasessential for determining precise functionality.
As parts of the control system were being moved fromthe old system to the new system there was a long period
where system users had difficulty in locating and properly
using the correct applications. There is a common problemof whether to use the new facility or the old. This can bevery confusing for users of the control system. Word ofmouth and special notes located in a book for such
purposes helped to ease this issue.
It appears that human nature is such that when afunctioning application exists, many people are reluctant tosee it change in form even if the new functionality is asuperset of the old. This may be nowhere more true thanfor accelerator operators. Accepting the wisdom of theseoperators, there is clearly some value in having therewritten applications appear very similar to their oldcounterparts.
2.6 Schedule and cost review
Software and hardware developments had been occuring
in a rather unorganized manner for a number of years prior
to the official funding of the Nova replacement program in
April of 1993. A very ambitious 3 year schedule wasestablished which identified the functions in each of theNova computers and a date when each computer would bedecommissioned. This schedule was followed and, with aslippage of approximately 8 months, the old computerswere fully removed by the end of December 1996.
In 1988/89 and 1989/90 there were hardware purchasesthat amounted to $265 K Can. and contributed toinfrastructural support. During the 1993/94 to 1996/97
period a total of $224 K Can. was budgetted for theUpgrade project, leading to a total of $489 K Can. spentfor capital expenditures. From 1988 to 1997 this wassupplimented from the regular Controls budget by an
unknown amount but likely in the range of $300 K Can.
In 1993 before the start of the project there were 3
people involved in central control system hardware
development and maintenance. At the same time there
were 11 people doing software development, maintenance,and administration, bringing the pre-project group total to14. There were no new hires or reassignments to assist
during the project. By December 1996 this total had been
reduced to 9 by attrition. During the project, those
involved retained their normal maintenance duties inaddition to any project responsibilities. Luckily, othergroups significantly reduced their expectations during thisperiod which allowed less time to be spent on regulardevelopments and maintenance, and thus permitted theproject to be completed.
3 Other completed projects
In addition to the removal of the Nova computers, otherprojects have been supported. Two examples of othercentral control system activities are the operator interface
work on the Proton Therapy Treatment Facility and aspectsof the beam extraction system for the new beamline BL2A.

3.1 Proton therapy
The Proton Therapy Treatment Facility is a facility fortreating eye cancers [2]. This installation delivers a beamof protons, via a primary beamline, to the patient locatedin a specialized chair in the treatment area. The operator
interface software and hardware, modelled on the centralcontrol system, was produced and is maintained by theControls Group. Patients are now being treated on aregular basis.
3.2 BL2A extraction
A radioactive beam facility is under construction atTRIUMF [3]. This facility will receive a beam of 500
Mev protons via a new beamline known as beamline 2A(BL2A). The BL2A extraction system, including thecontrols aspects, has already been completed [4].
4 New projects and directions
As the evolution of the central control system continues,
a goal will be to phase out all of the VAXes and replacethem by Alphas. This will provide better performance andsimplify system administration. As this happens, all of theCAMAC interrupt handling will be moved to the Alphas
and a transition to 64 bit addressing and operating system
support will be adopted. There are also a number ofenhancements involving
messaging, event handling,
displays, and other topics that are envisioned.
The remainder of the BL2A central controls will beimplemented and integrated into the central control system.
Related work will involve a link to the ISAC controlsystem.
Database activities are an important aspect of the central

control system. The existing X window interfaces toOracle will be supplimented and possibly replaced by web
based utilities.
Some web/database applications arealready in use and others are in development. A reliable,
three tier, database configuration employing Oracle, webservers, and web browsers is seen as a desirable goal.
5 Summary
The project to replace the aging Nova computers has
been successfully completed. The current configuration isrunning very reliably, providing good performance, therequired functionality, easy system management and fault
analysis,and flexibility for both maintenance anddevelopment. The operator interface for the Proton
Therapy Facility and the CCS controls for BL2A extractionhave also been completed. Enhancements to the existing
CCS and support for new facilities such as beamline 2A
are anticipated.
References[1] K.S. Lee et al, “Handling CAMAC Interrupts in Alpha
OpenVMS/PCI”, Proceedings of ICALEPCS 97,
Beijing, China[2] E.W. Blackmore et al, “Operation of the TRIUMF
Proton Therapy Facility”, Proceedings of PAC 97,Vancouver, Canada[3] P. Schmor et al, “Status of the TRIUMF ISAC Facility
for Accelerating Radioactive Beams”, Proceedings ofPAC 97, Vancouver, Canada[4] D. Morris et al, “The Control System for a New Proton
Extraction Probe at TRIUMF”, Proceedings ofICALEPCS 97, Beijing, China