USE OF LABVIEW AS LOCAL
SUPERVISION FOR THE LEP COOLING
AND VENTILATION CONTROL SYSTEM.
Ch. Bertuzzi, J. Fontaine, J.C. Frances, A. Guiard-Marigny, F. Josa, M. Vadon
CERN
INTRODUCTION
In the 80's the LEP/CV group was in charge of designing and installing the cooling, ventilation and fire
security equipment of LEP. Since the start-up in 1987, the accelerator's performance has increased steadily
and frequent upgrades have been needed. The circumference of the machine and the variety of technologies
involved make all systems very large and complex.
The ventilation process provides all underground zones (tunnel, technical zones, experimental areas) andsome of the critical surface buildings with conditioned air. It also provides local air cooling for electronics
and technical rooms as well as pressurisation of safe areas. Over 1.5 million cubic meters of air areprocessed every hour in 264 air handling units. The cooling and dehumidification functions are based onchilled water technology. The water is cooled by eight plants using centrifugal and reciprocating chillers.
When mixed it is also used at higher temperature (16°C) for electronic crate cooling. The heat generated
by the process is recovered in the air handling units, and the excess is discarded in a primary water loop.
Evaporative cooling towers with a total capacity of 220 MW dissipate the heat from these circuits andfrom several others such as the demineralized water loops designed to cool magnets, vacuum chambers,
accelerating cavities and experimental facilities. Finally, pumping stations supply all sites with water forcooling purposes and for fire hydrants. The underground fire security system sends alarms to the central
control room, interacts with other processes in case of emergency and provides extinguishing facilities in
the critical areas. New systems have been added every year since the initial design. Among the most recent
ones: compressed air generation for the cryogenic processes, desiccant cooling, logging of environmental
parameters for LEP Energy calibration purposes.

CONTROLS
In such a large and distributed system the controls architecture had to integrate turnkey equipment built by
many manufacturers. Matching both the control strategy chosen for the rest of the accelerator and thestandards used by industrial control suppliers was not an easy task. A clear limit of supply was set byasking the contractor to deliver fully functional equipment controlled by a Programmable LogicController (PLC), with an RS232 link using a well-defined protocol to communicate with the outside
world. All processes were therefore self-sufficient and can communicate with other equipment and themain Technical Control Room (TCR) via the LEP service network. Following the equitable purchasing
policy, CERN could not impose a PLC brand, and we ended up having most available models on themarket at the time (ABB, Siemens, Télémécanique, Texas Instruments,...). We did not think this diversity
to be an obstacle, since each contractor had to supply a local supervision console with a user friendly
interface. The link between the PLC cluster and the LEP network was done at CERN using an ECA(Equipment Control Assembly - G64 crate). A PC running LynxOS was chosen as gateway to the generalservices Token Ring network of LEP. The TCR used Appollos that were later upgraded to HP 700 series
workstations.
At the last minute, the local supervision consoles were removed from the contracts for budgetary reasons,
except for the chilled water process in four points among eight. This decision made commissioning andmaintenance more difficult. Only limited investigation tools were left at both ends of the data chain: theyet not fully operational central TCR system and the programming consoles. On one hand theinformation is event driven, summarised for efficient use by the non-specialised operators, who in case ofa problem try to restart the equipment and then call the experts if necessary. On the other hand
information is overwhelming and hard to interpret.

NEED FOR LOCAL SUPERVISION
The requirements for operation and maintenance were quite different from the ones of the TCR. Standard
packages do not include features such as trend and failure analysis, historical behaviour and processoptimisation.

The need for a specific tool became crucial since the performance of LEP was increasing rapidly, requiring
better control of the environmental parameters of the machine. The primary duties of cooling, heating anddehumidifying have been evolving towards a very tight control of all environmental parameters. In themeantime computing was undertaking major improvements, and one could legitimately expect processsupervision to be user friendly, to provide the user with a uniform man-machine interface (MMI) acrossall processes, and to be easily expandable.
After a comprehensive analysis of the market, we came to the conclusion that PLC manufacturers kept
selling supervision systems that were closed, expensive and platform dependant. More suitable for ourapplication were general purpose supervision tools such as Factory Link, LabVIEW, In Touch, PC Vue,
… which have key advantages: easy connection to a wide range of PLC’s, availability on genericplatforms and operating systems, and lower price.

CHOICE OF LABVIEW
The co-ordination of industrial controls is organised at CERN [1], and in the fall of 1992 when ademonstration of LabVIEW took place we thought this product would be most suitable for our purpose.
The wide acceptance of LabVIEW, its cross platform portability and its low cost influenced our choice.
Even though LabView was first presented as a laboratory oriented package, it has evolved towards being ageneral purpose compiler that includes many advanced libraries and still remains usable for simple tasks
by a non-programmer.
At the time of the last ICALEPCS conference, in Berlin, our software and hardware choices were settled.
Since then, the system has been developed and become operational. The present paper presents our firstremarks and draws some conclusions after several months of continuous operation. Although certainly
subject to future enhancements, it has already proved to be an invaluable tool for monitoring the coolingand ventilation processes of LEP.

INTEGRATION IN THE EXISTING CONTROL ARCHITECTURE

TechnicalControlRoomRemoteOperation
X Terminal

LEP Token Ring

HP StationCERN Ethernet

ServerEven pointof LEPLabVIEWPCA
MIL 1553

RS232

ECA
RS232

ModemFire
security

Electrical Secondary
alcovescircuits

PLC

Coolingtowers

Siemens

Field Bus
ABB

Télémécanique(April)Texas Instruments

Securiton

Concentrator

Chillers

OperationTeam

Air handling units

Existing controls
The HVAC controls for LEP were designed in the 80's, but they follow closely what is now commonly
accepted as the standard architecture.

BeepSince most of our systems were industrial in type, we used the ECAs as bridges between the RS232
coming from the PLC's supervisor and the Mil1553 bus. To guaranty fast data access through thenetwork, each ECA keeps in memory an updated image of the data contained in its dependent supervisors.
It also translates the commands issued by the control room into the proprietary protocols used by PLCs
(JBUS, XCOM, 3964).
Connection of the local supervision
Three solutions were considered for the integration of the local supervision console into the existing
control architecture: connection to the PLC field buses (supervisor), to the Mil1553 (ECA), or to thetoken ring (PCA).
The first solution delivered excellent performance but required the installation of new cables, and imposed
the coding of all PLC drivers within LabVIEW.
The simplicity of the second solution was very attractive. We immediately thought of using an ECA as
an RS232 - Mil1553 bridge. The ventilation ECA matched well since it had a spare serial port, could
handle the additional load, and was located in a convenient area. We also decided to keep the syntax andformat used by Mil1553.
The third alternative inserted another layer with no gain over the second solution.

DEVELOPMENT OF THE SUPERVISION PROGRAM
Data handlingAll data are read in 16 bit format through the serial port and stored in memory every thirty seconds, which
is more than adequate for our slow processes. The data being stored locally, all kinds of processing werepossible. The purpose of our application being to facilitate the work of the operation and maintenanceteams, weekly meetings allowed examination of the design and its matching to the real needs.
We were urged to design a supervision console based on the traditional animated synoptics representing
the status of the controlled equipment in real time, coupled with alarm printing. This was quite simple,
since LabVIEW contains many graphing tools and imports any image created by drawing programs. The
lack of a technical symbol library was not a real drawback, and we rapidly created our own.
However, we soon realised that implementing this concept developed primarily for central control rooms
did not fit our needs. Trends, historical behaviour of the equipment and failure analysis were missing
features that were important to us. Hence we added a one hour ring buffer to navigate back and forth intime to replay specific sequences, as with a video cassette recorder. This system was praised and we soon
were asked to increase its capacity to days, then months and finally years. We eventually replaced the ringbuffer approach by an extensive data handling program.
Data are stored into daily files constituted of 1440 fixed length records filled minute by minute. This rigid
scheme minimises access time but is not space efficient. Consequently, we use a compression utility(Stuffit) to automatically recover disk space every night and transfer the stuffed file to a file server forarchiving. We also use a cache mechanism to limit the wait induced by file expansion. This technique isworth the effort since the compression ratio exceeds 95%.

Synoptics

The bulk of any supervision system consists of animated synoptics. We had to develop hundreds of themfor our application. They are organised in a tree structure, from overall snapshot to zooms on details, butthey are nonetheless completely independent. Each synoptic is a self running program that gets its datafrom a global variable, and thus a bogus code cannot corrupt the rest of the application. This deliberatechoice allowed several persons to develop simultaneously without constraint.
A template was first created. It is an empty front panel associated with computer code that contains both asynchronised loop triggered by the main program when new data arrive and a fast loop handling button
and menu events. In addition, we developed a virtual instrument (VI) library containing standard
subroutines that perform often used bit array operations, data conversion, threshold detection, …
After a one week course the persons in charge of the processes, who are not programmers, were able todesign the user's interface and its associated logic. The only challenge was to enforce a common strategy
to obtain a uniform MMI for all processes. This was done during "lively" weekly meetings.

Alarm system

First, we decided to print alarm messages on paper as they occurred. Once again practice showed that thistraditional technique was not adequate for our use, since alarm lists were inevitably discarded or lost. We
then decided to store the messages for on-demand printing, but users complained that when a general alarmoccurred the system would spill hundreds of them simultaneously, losing the chronology. The initialproblem, such as a 400 kV power supply failure, would be hidden. Instead, we created an alarm chart
showing the status of each system versus time. At a glance, it is easy to check that systems have
performed well during the previous few hours. By the means of a navigation cursor it is possible todetermine the exact time of a fault, and then call directly the appropriate synoptic and use the VCR
navigation tool to replay and analyse the animated sequence. This chart is now the main investigation
tool for our maintenance crews.
During business hours people are present on the points of LEP, so we decided to install modems that
automatically page a member of the maintenance team in case of an alarm. We can generally intervene
before a major failure.

Charts

At the beginning of the project charting capabilities had a low priority on the wish list, but after a year ofoperation they represent the most popular feature. It is now accepted that a good curve is worth more thana thousand alarms. Due to insistent requests we even had to trade our black and white printers for colourones.
In most supervision packages the designer must define a limited set of historical values. In contrast oursystem can plot any data for any period of time since everything is stored on the hard disk. When callingthe standard graph module the synoptic passes a variable containing all information: offset, scaling
factors, legend, … . This yields the greatest flexibility.
Data and alarm generation
Operators often complained because alarms would trigger emergency stops without prior warning. The
absence of pre-alarms was unfortunate, but was justified by the limited amount of memory available on
our PLC. Similarly, useful calculations such as power usage and thermal balance were never performed.
We added a module in our program for this unique purpose.
Overall system optimisation
The HVAC systems are physically interdependent and the accelerator relies on them. For example, afailure of the primary water loop soon provokes the interruption of all equipment cooling, leading tobeam loss. In such a context it was unfortunate that the interactions between control systems remainedlimited.
A first attempt at such an interconnection was made with the dipole temperature control. We made apermanent connection to the magnet temperature measurements, and were then able to adjust thedemineralized water loop set points as a function of dipole temperature to guaranty a stability better than
±0.2°C. This application demonstrated LabVIEW's remarkable flexibility in an accelerator environment.
This approach could now be extrapolated to other processes for overall system optimisation.

DEVELOPMENT ORGANISATION
Ventilation

Physics

Alcoves

Data acquisition
Alarms generation (beep)
Vues synchronization
Historical data managementChilled water

StoragePrimarywater

Mixed water

Alarm graph
Analog values graphs

Low conductivity water
Raw water

Since the start of the project we have been meeting on a weekly basis. We first discussed data acquisition,communication protocol and hardware characteristics. Rapidly the overall supervision structure wasdefined, and we could start the design of the synoptics. We thought that the MMI definition would be aneasy task, but we soon discovered that colours, symbols and naming conventions could generateemotional debates. Compromises were eventually obtained on almost everything, sometimes painfully,
but we never reached an agreement on details like background colour …
Later we started inviting people of the maintenance team, who gave us valuable feedback. They played an
important role in testing beta versions and criticising the design. Their sensible remarks often influenced
the course of the development. However even the earliest versions allowed one to diagnose faults in theHVAC systems.
Every time we set-up a point of LEP, we would invite its maintenance team to the meetings. Gradually
the topics were more oriented towards the exploitation of the installations. Today the meetings are
entirely dedicated to the analysis of the past week's history and the refinement of the intervention strategy.

COSTS AND RESOURCES
The hardware acquisitions consisted of Macintosh Quadra 800 machines (16 MB RAM, 240 MB HD)
equipped with 17" monitors, eight ink-jet printers, four modems, and a 1 GB hard disk for archiving.
After an initial payment of $2000, the use of LabVIEW only required the payment of a $500 annual fee tobenefit from the maintenance and support program. Runtime systems were free of charge.
The direct human resources represented two man-years, but the induced work was substantial. Processtuning, PLC program debugging and systems modification were not negligible tasks. We should stressthat this work was accomplished in addition to people's normal duties and that no extra resources wereused.

ESTIMATION OF THE PAYOFF
Debugging
The local supervision was a mere prototype when we first observed some unstable cooling system
operation. We found that in some PLCs, A/D converters would poorly truncate analog values. We also
discovered that some initialisation subroutines were causing unreliable start-up of equipment.
The processing load added by the supervision unveiled serious problems in the ECAs that were formerly
occurring occasionally. Gradual buffer fill-up, communication freeze-up and several other bugs weredetected and fixed.
The synoptics were developed on line. This permitted constant interactive check of the whole dataacquisition chain, from the sensor to the screen displays, together with the validation of the Oracle
database information.
TuningBesides facilitating daily operations, tuning revealed flaws in our systems. For instance PID parameters
had not been adjusted by the firms because it required too much time.
On the other hand we had to exonerate networks, which were blamed for all problems. Inter-point
communication was established only long after the LEP start-up. The heterogeneous architecture did notease exploitation. Hence our system kept a bad reputation, but when all unrelated control problems weresolved the accusations vanished.
The local supervision system acts now as a referee between dueling persons. Since process interactions
cannot be misinterpreted, the debates are always constructive.
New investigation techniques
After a short learning phase the operation and maintenance team was capable of performing periodic
checks of the main systems to prevent problems. Later, when the alarm system and its associated paging
became operational, they could react even more promptly.
Recently we started using curve analysis to implement conditional maintenance. By observing evolutionof the duty cycle's frequency, we were able to detect minor leaks in LCW circuits and refrigerant losses inthe electrical alcoves' air conditioning units. Since both places are not accessible during LEP operation,
we planned future interventions in agreement with the physicists. We then could fix the problems in atimely and cost effective way.
In the past, considering the limited investigation tools, the TCR would have called the people on duty
indicating that a major leak had occurred in the tunnel. Driving distance between LEP pits, night-timeintervention, customs procedure for spare parts transit, and … Murphy's law could lead to a longinterruption of physics.
This new active approach is priceless and has reduced dramatically LEP failures due to HVAC systems.

CONCLUSIONSIn a timely way, with limited budget and resources, we designed the local supervision system that was
missing for the LEP cooling and ventilation processes. The choice of LabVIEW allowed nonprogrammers to write most of the code and develop specific features to suit the needs of the operation
team. The initial goal was rapidly exceeded, and this system now acquires 30,000 points of data, storesthem at one minute intervals, and features synoptics, graphs, alarms and long term archiving.
This accomplishment allowed the full debugging and tuning of the controls of the LEP cooling andventilation equipment. Moreover, by motivating the personnel and the operation team, this successfulproject triggered a change in mentalities and established a new style for system operation.
References:[1] Interfacing industrial process control systems to LEP/LHC,M. Rabany , Icalepcs conference Tsukuba
91, Proceedings p269-273