EXPERIENCE IN THE EXPLOITATION OFA LARGE CONTROL SYSTEM
G.Daems, F.Perriollat, Ch.Serre
PS Division, CERN, CH-1211 Geneva 23, Switzerland

Abstract
Experience of a four-year exploitation of the large control system of the CERN PS accelerator complex ispresented with special emphasis on the parameters which are very sensitive to the exploitation teamproductivity. The software tools suite used in this daily maintenance is described and a particular analysis of thepower and benefits of the advanced software technology used for the architecture of this suite is explained. Theintegration of this suite into the control system is presented, as well as its use in the control systemdevelopment phase. Some considerations of the potential benefit of an Object Oriented equipment access areoutlined.

1. INTRODUCTION
The latest version of the CPS (CERN Proton Synchrotron) control system has now been in use for four years(1992- 5) on six of the nine accelerators forming the CPS complex [Fig.1]. The system [Fig.2 & 3], [1]is based on the “Standard” model with 3 levels: interaction, front-end computing and equipment control. An
Ethernet network running TCP/IP links the user interface, the central services and the real-time equipment
processing, while field-buses (CAMAC and 1553) interlink the equipment to the real-time equipment
processing. The total number of items controlled is around 10000, not including the instrumentation.

Fig. 1: Layout of the PS Complex
During these four years, the exploitation of the control system has been carried out in parallel with the old one
to keep the complex in running order. By exploitation we mean the maintenance, problem solving andimprovements to the control system. The people charged with this exploitation have had to adapt theirknowledge and tools to the new system in which they also played an active part during its development.

The experience gained in the exploitation of such a large control system is presented, with a special emphasison the integration and the expected benefits of modern software technologies such as the Object Oriented
Programming and Development and the Knowledge Based Techniques.

2. WHAT THE CPS CONTROL SYSTEM EXPLOITATION INVOLVES
2.1 The environment
The CPS Accelerator Complex operation runs round the clock, 24 hours a day, ten months a year, with twoshort intermediate stops of 2 to 3 days. This means that new installations, modifications and maintenance are
not easy. Nevertheless, because the CPS complex is the source of all the particles beams at CERN, its operationevolves continuously. There are several machine development periods interlaced with the normal running of themachines which result in constant modifications and updates to the control system and may require softwarechanges in the operational environment. In addition two points should be noted; the software developers often are
only transitory people such as fellows or students, with all the follow-up problems and the exploitation teamhave to deal with a historical inheritance, translated into a great diversity of interfaces and equipments, some ofthem only partially renovated due to budget constraints.
The CPS operation requires frequent daily operational changes, for example for antiproton transfers. The present
staff policy results in a continuous rotation of operators with short-term contracts which leads to lessexperienced staff. As the operation of the machines is seen by the operators through a thick control layer, it isnecessary for controls people to have some knowledge of how to operate in order to be able to help theoperation crew in its daily job. This sometimes allows the exploitation people to identify as operation faults,
things that were initially reported as control faults.

Fig. 2: The standard architecture applied to the CPS complex
2.2 On-line and Off-Line exploitation
The control system exploitation is divided in two types of work, the on-line exploitation, which is theimmediate answer to faults or wrong conditions in the control system elements (hardware or software) and theoff-line exploitation, which can also be called the follow-up and the solving of problems encountered during theoperation of the machines.
The on-line exploitation is the field of a small and well experienced team which take charge of all problems andbad behaviour labelled by the operators as “control faults”. People of this team trace the faults, analyze andcorrect the faulty control elements (hardware side), check and restart the faulty software components and finally
support the operation team to recover and return to the required operational state.
The off-line exploitation involves practically all the people of the Controls Group in order to improve thereliability and the robustness of the control system (hardware as well as software). A regular weekly exploitation
meeting enforces a strict follow-up of the different problems and changes. Agreed upgrades, extensions orenhancements are mainly coordinated by the exploitation team. It also proposes and develops the necessary
diagnosis and exploitation tools.

SL/control network

PS/control network
Common PS/SL database
Meyrin Control Roomand central services(Central Timing, SOS).

SL FDDI backbone
DSC

DSC

DSC

Local serversRouter

DSC

DSC

DSC

DSC

DSC

DSC

DSC

DSC

DSC

DSC

CERN/public network
CPS

bridgeCERN FDDI backbone network

Local server
PCs

PS offices and tests

PSB

bridgeUnix workstations

DSC

DSC

DSC

Local server

Unix servers

DSC

ISOLDE

PCs
DSC

LINAC 2&3

bridgebridge
DSC

DSC

DSC

Local server

Novell server

LEAR

LPI
bridgeDSC

DSC

Local control room
stations & accelerator
specific front-ends

bridgeDSC

DSC

VMS server

DSC

DSC

Local server

Fig. 3: The CPS Network
2.3 The Exploitation Team ?
The exploitation team is a small team with good experience, an overview of the system and general purpose
knowledge and its first priority is to keep the control system. running correctly, whatever the time and the workrequired. A “piquet” service (5 persons) is in charge of the on-line services; one person is assigned for one week
(24 hours a day) to answer as fast as possible the calls from the operation crew if there is any breakdown of thecontrol system components.
This exploitation team provides a single entry point (on-line and off-line) for all the requests concerning theproblems, improvements and modifications to the running control system, which enforces directly thehomogeneity of the hardware/software solutions. The persons of this team are attached to the other sections of
the control group to develop diagnosis and exploitation tools. This is mainly software for interface andequipment access and for the overall survey system.

3. EXPLOITATION TOOLSThe performance of this team depends on the continuous education of the persons, the documentation of thehardware and software components of the control system, the communication between the developers and themembers of this team, but also on the tools provided to this team to be able to diagnose, recover, restart and/orset up the components of the control system. Data logging of the element values, easy-to-use links with the database (ORACLE), automatic configuration of front-end processor devices from the data base, tracing and log offaults with post mortem analysis capabilities [2], testing and setup facilities are the main tools needed by themembers of the exploitation team to fulfil correctly their first priority job.
The status of the different controlled elements of the CPS complex are permanently monitored on the screens ofthe Control Room workstations, in a window generated by the Alarm process [3]. This program, used by boththe operation and exploitation crews, scans the status of the controlled equipments every 30 seconds, via theuniform equipment access level (Equipment Modules [4]). In addition, it surveys the VME and CAMAC crates,
both for overall hardware and software status.
This Alarm process is the main entry point for diagnosis tools and information. It allows the presentation of :- the details of faulty equipments- the layout of input/output crates (VME and CAMAC), the modules in the crates and the equipments controlled- the front-end processor status : accessibility, interrupts and errors- the fault log, the pulsed status and the interrupt layout of the timing system elements.
From this information one can reset a faulty equipment (such as a power supply, a vacuum pump, or a RFcavity) or detect a fault at the element level. A front-end processor not accessible though the control system canbe remotely rebooted after verification of its status. The history of the reboots and the errors are logged and canbe displayed on request.
Most of the exploitation tools can be called through this Alarm process, either directly or through two
important general programs :The first program, Equipment Info, gives the possibility of selecting dedicated programs to diagnose, test
and initialize an equipment or a control interface; this is especially the case for the different instrumentation andfield-buses (CAMAC and 1553). One can also test a particular equipment access and initialize it if necessary.
Special debug or repair commands, documentation concerning the Equipment Module and the layout ofsubsystems (timing systems for example) are also available from this general panel of services. From an otherspecial panel, an equipment can be accessed through the Equipment Module either by its name or its number or
its address. The selected equipment can be tested or controlled up to the last bit.
The second program, Setup, is created to reinitialize automatically an equipment or a whole system (aCAMAC crate for example). It is a very important and general purpose tool which is described below.
A few other important tools exist, which can be called for a specific purpose. These are:- for the verification of the description and synchronization of the different operations of the CPS accelerator
complex, where a rule-based consultant can be called to verify if the required schedule is acceptable. This is
the Beam Card Desk Checker [5]- for the automatic configuration of the software of the front-end processors directly from the data kept in theORACLE data base [Ref.6]- for the Data Browser/Editor which gives the possibility to read, compare and modify the operational values
stored in the data table of the equipment access modules (Equipment Module); with this tool, the exploitationteam can work on the actual operational data, the regularly-saved values and the values called references.
4. SETUP : KNOWLEDGE-BASED TECHNOLOGY
The development of the Setup program is based on an Object Oriented programming, together with a proceduralknowledge representation [7]. Setup provides the means of initialization and of non-destructive testing for theaccelerator equipment based on CAMAC and VME control interfaces. It is a rule-based process which can copewith any unknown initial state of the equipment.
4.1 Utilization
This Setup program is used both by the exploitation and the operation Teams. The primary uses are after a
power failure, a shutdown or the replacement of a faulty hardware module or crate; Setup allows one to initialize

the hardware with the correct procedures and the correct operational values (last saved values). Setup is also used
to reset a faulty equipment (power supply for example) to come back to the normal state. When an
instrumentation gives wrong information or does not run correctly, the best solution is to call Setup. Finally
Setup can also test an equipment without disturbing or with the minimum disturbance of its normalfunctioning.
4.2 Main points and Realization [7 ,8]
The Setup system can easily be adapted to the control system environment. It is flexible and evolutive andadapted to a huge variety of hardware/software modules, including the treatment of hardware-coupled equipments.
The description authorizes different sets of objects, so that the Setup actions can be process- or hardwareoriented. The access to the control system elements is made using the uniform equipment access method(Equipment Modules).
During the running of the program, the user is informed about the current state of the equipment and about thediagnosis of the procedure in real-time; if an action is not successful, the diagnosis of the faults is given. Post
mortem analysis exists for the Setup protocols that have taken place during the test or the reset of an equipment
or a process.
The realization of the program is based on the development of an object model which reflects the structure of theControl system equipment access: front-end processor, CAMAC loop, CAMAC crate, CAMAC module andEquipment. The knowledge-based representation gives the description of object classes, control rules andoperation algorithms. The concrete object lists are automatically “instanciated” from the Real-Time DataBase. The realization of the Setup program is based on an expert system shell (PROSC: Procedural Reasoning
Object System for Control) which allows object oriented knowledge description, procedural reasoningtechniques, real-time features and direct queries to the ORACLE data base.
4.3 Benefits for the exploitation of the control system
The user interface of Setup is done through X-windows/Motif panels which are called by the Alarms process, theexploitation programs or the hardware surveillance at the level of the faulty object. The Setup facility is also
accessible from terminals using a certain number of commands (for tele-diagnosis capability).
As stated in the previous paragraph, the Setup rules and algorithms are easy to describe and to change for thedifferent classes of the control system. This results in a well-adapted, up-to-date and reliable tool that executes
the necessary actions demanded by the operator, taking into account real-time events and hardware-coupled
equipment, which are otherwise invisible on the interaction level.
The automatic generation of the concrete object lists related to a “super” object to initialize is one of theessential benefits of this setup. The Setup software executes automatically all necessary actions, either when one
has to replace one hardware module or to initialize one equipment or a whole CAMAC loop. The setup can becalled either from a hardware description layout or from a software equipment level. The Setup operates today on:- 100 Front end processor VME crates (DSC)- 130 CAMAC crates- 3000 equipmentswith more than 100 object classes and 200 control procedures.
It is a necessary and powerful tool for the daily operation of the six machines of the CPS complex controlled by
an example of the standard model of control system for accelerators. The technology in use and the integration inthe control system provide the capability to evolve with the system without difficulty.
5. CONSIDERATIONS ON OBJECT ORIENTED TECHNOLOGY WITH BENEFITSFOR THE EXPLOITATIONThe equipment access software structure of the CPS control system has been Object Oriented since 1988 [4].
The previously realized Equipment Module model has proved to be very useful for the exploitation anddevelopers because it is a standard generic facility which gives access to all the process equipment on a standard
way and allows the use of powerful and generic tools.
We already found that Object Oriented Techniques facilitate the tasks of the exploitation Team by providing:- easier maintenance of software programs (produced by temporary or external people)- faster and safer modifications of the software products

- easy integration of jobs developed outside the site (to benefit from external expertise)- good integration with an ORACLE Data Base description- independence of the control system architecture, to be able to follow the technological evolution of thedifferent control system components.
The fields in which we are less confident, due to our short experience, are on the benefits to be gained fromobject languages. Especially, we have some doubts on:- the amount of time necessary for a large community of software developers to be fluent in C++ and in theobject oriented technology environment- the difficulty to define and setup the object libraries for the different usage- the complications in building a coherent set of abstract object models for the various control process provided- the facility for the casual software developers to use efficiently the object oriented development tools toincrease substantially their productivity- the impact of Object Oriented Databases and their level of standardization- and, last but not least, the stability of the emerging standardization, especially on standard C++ and thebasic object library (both on real-time applications and on object broker technology in a distributedenvironment)
All these points can have a direct impact both on the long term exploitation (and re-use) of the huge amount ofcontrol software, and on the easy integration of third party components.
6. CONCLUSION
One of the main tools used by the exploitation team on the large CPS control system is the Alarms process,which acts both as a surveillance program running continuously on the operational work stations and as aswitchboard giving the possibility of calling the other exploitation tools described in this paper.
The other important tool is Setup, which uses the object oriented and the knowledge based technologies to offer
a basis for the test, setting up and initialization of the different control system elements, both individually atthe equipment level and at the subprocess level. These technologies could be used for other applications such as
the timing survey, synchronization generation and the interpretation of complex results. These technologies haveshown their capability in solving specific problems and they are able to be integrated in a classical system in avery cost effective way.
Our long experience in use of the Object Oriented Technology gives us full confidence in the future benefits oflarge scale implementation. However, to be in a position to carry out efficiently the large scale integration ofthis type of software, we believe that more time is needed and that wider open collaboration and exchange ofexperience is essential. These points should receive the full support of the management of the major projects in
our field of activity.

7. ACKNOWLEDGMENTS
We give special thanks to the members of the exploitation team whose efforts kept the control system runningwith a high level of availability (< 1% of machine down-time had been due to control problems). We are also
grateful to the Control Group members who continuously try to provide excellent exploitation tools using themost suitable and efficient software technologies.

REFERENCES:[1]F.Perriollat, Ch.Serre, “The new CERN PS control system, Overview and status”, ICALEPCS, Berlin,Germany, October 18-23, 1993, Nucl. Instr. And Meth. A352 (1994) 86.
[ 2 ] M.Beharrell, G.Benincasa, J-M.Bouche, J.Cuperus, M.Lelaizant, L.Merard, “Model based, detailed fault
analysis in the CERN PS Complex equipment”, ICALEPCS 95, in these Proceedings.
[ 3 ] J-M.Bouche, J.Cuperus, M.Lelaizant, “The data driven alarm system for the CERN PS accelerator
complex”, ICALEPCS, Berlin, Germany, October 18-23, 1993, Nucl. Instr. And Meth. A352 (1994) 196
J-M.Bouche, “Utilisation du programme Alarm”, 6.03.95, PS/CO/Internal Note 95-17.
[ 4 ] L.Casalegno, J.Cuperus, Cl-H.Sicard, “Process equipment data organization in CERN PS controls”,
ICALEPCS, Vancouver, Canada, October 30-November 3, 1989, Nucl. Instr. And Meth. A293 (1990) 412.
[ 5 ] J.Lewis, P.Skarek, L.Varga, “A rule-based consultant for accelerator beam scheduling used in the CERN
PS complex”, ICALEPCS’ 95, in these Proceedings.
[ 6 ] J.Cuperus, A.Gagnaire, “Automatic Generation of Configuration files for a distributed Control system”,
ICALEPCS 95, in these Proceedings.
[ 7 ] J-M.Bouche, G.Daems, V.Filimonov, V.Khomoutnikov, F.Perriollat, Yu.Ryabov, “Representation andUsage of knowledge for Initialization of accelerator control equipment”, presented to RT-95, East Lansing,MI USA, May 23-26, 1995.
J-M.Bouche, G.Daems, V.Filimonov, V.Khomoutnikov, Yu.Ryabov, “Knowledge based Setup facility”,
22.11.94, PS/CO/Internal Note 94-81.
[ 8 ] G.Daems, V.Filimonov, V.Khomoutnikov, F.Perriollat, Yu.Ryabov, P.Skarek “A knowledge based
control method : application to accelerator equipment Setup”, ICALEPCS, Berlin, Germany, October 1823, 1993, Nucl. Instr. And Meth. A352 (1994) 325.