Integration of Industrial Equipment and Distributed Control Systems
into the Control Infrastructure at CERN
R. J. Lauckner, R. Rausch
CERN, 1211, Geneva, 23, Switzerland.
Abstract
CERN Accelerators and Services have an existing large
investment in control networks, fieldbuses, front-end
process computers, workstations, system software andapplication programs. Increasingly, industrial equipment
and complete Distributed Control Systems (DCS) arebeing installed to rejuvenate old and obsolete systems andnew projects for LHC are taking shape.
The object of this paper is to report experience of
CERN′s control engineers on integration of industrial
equipment and systems. Several projects are describedcovering a vast area of applications.
The paper presents several interfacing issues such as
network connection of industrial equipment, software
integration, homogeneous equipment access and alarm
retrieval compatible with the existing Accelerators andServices Control System.
1 Introduction
For CERN′s Accelerators and Technical Services anincreasing number of industrial systems is being installed
to rejuvenate old and obsolescent systems. Several
projects concern the SPS accelerator which will be theinjector for the future Large Hadron Collider which will
start operating in 2005.
In addition, many new systems for the LHC are entirely
specified from the beginning of the project to use
industrial equipment. The control equipment concerned
ranges from individual Programmable Logical Controllers(PLCs), through clusters of PLCs interconnected by afieldbus, up to complete DCSs delivered to CERN as
turnkey systems with their dedicated supervisor included.
These new industrial control components may or may
not, need to use part of CERN′s existing communication
infrastructure. The technical requirements of these industrial systems upon CERN′s control networks, and viceversa, must be identified in terms of Response-Time,
Bandwidth, Protocols, Availability and Reliability.
Integration of industrial software is another important
issue. Depending on the level of the controls architecture
at which industrial equipment is connected, the communication with pre-existing equipment poses different problems and challenges. CERN′s control engineers have toprovide the operators of the accelerators and technicalservices with a homogeneous man-machine interface and acoherent treatment of alarms generated by the various
industrial systems and the existing control system.
To buy and to contract for industrial control equipment,
it is essential to know beforehand what products manufacturers are offering and what level of expertise is required
from the supplier. The technical specifications related tothe project must be written following standards andmethodologies commonly in practice in industry.
Finally, to follow-up, to supervise the execution and tocomplete successfully industrial contracts requires theadhesion of both parties, CERN and the contractor, to arigorous project management plan.
2 Typical applications of industrial systems
To illustrate the diversity of industrial systems installedat CERN, operational today or under development, we
present some recent and significant projects. We describe
the SPS Machine Access System, the CERN Site Access
System, the SPS Smoke Detection and Tunnel Ventilation,
the Interlock System of the SPS Main Magnet Power
Converters and the Modernisation of the Beam Target
Electronics.
2.1 SPS machine access system
The SPS machine access system covers both the accessto the accelerator tunnel and to the North and Westphysics experimental areas. The access to the SPS tunnel
is controlled by the machine operator while the experimental areas accesses are prepared and terminated by thephysicists themselves by running programs from beam
control terminals in their experimental barracks. These
programs monitor the required critical elements to satisfy
the conditions for access and then send the necessarycommands to the door to allow entry. It is therefore
important that the machine access system for the experimental zones is able to communicate with the machine
operator and also with the tunnel access system directly.
The SPS machine access system uses the multi-protocol
communication capability of the controls network.
Three industrial PLCs and a HP-UX workstation are
directly connected to the network; the PLCs control theaccess hardware, keys, film badges, grids, displays while
the workstation runs the Factory-Link Supervisor andcommunicates via the Sinec H1 Protocol with the PLCs.
The HP-UX acts as a gateway between the operator′s X
Terminal server and the PLCs [1].
2.2 CERN site access system
A CERN-wide site access system is being installed inthree phases, first at 12 sites around the SPS Accelerator,
then at 10 sites on the Swiss Meyrin area and the next
phase will be at 17 sites around the LEP Collider.
Each remote site communicates by TCP/IP over thecontrols network via a local HP-UX computer that runs aproprietary supervisor and acts as a gateway between thelocal cluster of PLCs and the central access control room.

At a site, the local PLC cluster interfaces to gates, doors,badge-readers, lights, etc..
Each site is controlled remotely, but is also capable ofoperating autonomously in stand-alone mode, should anetwork problem occur. To allow local operation each siteholds a subset of the central database of persons authorised
to access a particular site.
At the central control room one finds HP-UX workstations, X-terminals and a global access server. The site
access system consults a central CERN personnel database, to obtain individual access authorisation. Alarms,
generated at the local sites, are sent for analysis to thecentral alarm server and for presentation to the site access
operator.
2.3 SPS smoke detection and tunnel ventilation
A global smoke detection and ventilation system isbeing installed in the SPS tunnel in order to detect fire andto activate the tunnel ventilation system according to apredefined strategic plan.
Each of the SPS tunnel arcs contains a number of smokeand fire detectors linked to a local PLC; on top of the tunnel pits are ventilators, motors and fans controlled by asecond PLC. The tunnel PLCs are connected via aProfibus fieldbus to a master PLC located in each of the
SPS surface buildings. Seven master PLCs will be directly
connected to the controls network and the communication,
between these master PLCs and the HP computer runninga supervisory system under HP-UX, is done by a TCP
protocol (Send/Receive interface, Transport Layer 4, RFC
1006).
In each SPS building and on top of the local tunnel pit
there is a local synoptic display for the firemen depicting
the status of the fire detection and ventilation systems.
This local display is connected directly to the master PLC
and consists of an industrial PLC from which full local
operation of the ventilation system is possible. Remote
operation and visualisation of the status of adjacent sites isalso feasible when the network is operational and notaffected by the fire. The global supervision is done fromthe firemen′s central operation control room.
2.4 SPS main magnet power converters
The control and interlock system of the main magnet
power converters is obsolete and its replacement by a new
one is necessary. The new interlock system is based on atechnical infrastructure implemented with PLCs connected
to a Profibus fieldbus.
The SPS main rectifiers stations (14 dipoles and 3
quadripoles) are fed from an 18 kV substation located in
the central electrical building. Cables bring the power andthe remote control to the individual rectifiers. The
distributed power converters around the SPS accelerator
could imply considerable risks for the personnel andmaterial. The control and interlock system detects all
faults and may stop all the main power converters or, if it
is necessary, switch off all 18 kV circuit breakers. Faults
are displayed on the front panels of the individual
converter and the information is transmitted to the beam

control room. In addition, the control and interlock systemallows the remote control of the power converters fromthis control room.
The new control and interlock system is composed of 18
local PLCs. One PLC for each converter manages the local
faults and communicates through a Profibus fieldbus withthe master PLC. Another PLC manages the 18 kV substation and reports local faults. The master PLC supervises
the entire system, gives the real-time information from any
part of the installation, manages the common parts of themain power converters and communicates with the HPUX based gateway. One Profibus fieldbus inter-connects
all the 19 PLCs installed around the 7 km of the SPS ring.
The HP-UX workstation runs a proprietary supervisor andacts as a gateway between the master PLC and the CERN
controls network. This computer allows the project to be
integrated into the architecture of CERN′s accelerator andservices control system. The implementation of the manmachine interface is based on Motif and X11.
2.5 Modernisation of the beam target electronics
The aims of this project are the re-engineering of the
SPS target electronics and associated controls and its
integration into the existing CERN accelerator controls
architecture. The objectives are: to replace obsolete homemade electronics, to homogenise equipment controls withindustry standards, to carry out the project in collaboration
with industry, to specify a configurable, modular, data andevent driven solution, to provide low level equipment
control and surveillance software, to specify theapplication interface layer, to integrate the equipment
controller within the SL-EQUIP & SL-ALARM standard
software packages [2,3] and finally to maintaincompatibility with the present application, operation andexpert programs.
This project, being the most recent one, will also be themost ambitious. It will make use of a new family of PLCs
on the market that use open communication standards. A
collaboration of equipment group experts and controls
engineers will study and define a generic approach for
integration of industrial systems in CERN accelerator
control system.
Manufacturers are beginning of offer open PLC productswith TCP/IP communication. For this project the new
family of SIEMENS PLCs, type S7-300/400, has been
selected. The challenge will be to integrate these PLCsfrom the network and from the equipment access point ofview in an homogeneous fashion compatible with theexisting infrastructure. We are considering porting CERN
communication S/W into these PLCs or providing acommunication gateway in between the central workstations and the cluster of local PLCs. Another technicalverification is to check the bi-directional communication
over multiprotocol network routers.
3 Interfacing issues
For many years CERN has adopted TCP/IP as thecommunication protocol among workstations, servers andfront-end computers. Accelerator and Services Controlsare fully implemented in the UNIX environment while thefront-end computers run a UNIX compatible real-timekernel.
This has not been the case for industrial systems until
recent years. Industrial systems need their own private
network, run their special communication protocol, havetheir own real-time kernel and operating system, use their
dedicated database and offer their proprietary controls andmonitoring supervisor. All are justified for maximum
performance, optimum investment and, to some extent, tobe different from the competition.
3.1 Network connection of industrial equipment
The connection of PLCs into an existing controls
architecture can be done either at the network level or at
the fieldbus level. The choice between these two options ismade taking into account several factors: the communication openness, the inter-operability among different systems, the number of connections and their geographicaldistribution, the bandwidth required, the network protocol
and the equipment control protocol to be used. The present
experience of CERN′s control engineers takes into accountthe evolution of the multi-protocol networking technology,
the acceptance of TCP/IP for the communication with
Programmable Logical Controllers (PLCs) and the emergence of new open DCSs proposed by manufacturers [4].
3.2 Homogeneous equipment accessWe discuss two aspects of equipment access. The first
concern is the operator and end-user of the equipment and,in order to provide a good solution in that domain, theprogrammers′ equipment access interface is also important.
The previous generation of industrial equipment employed at CERN was supplied with a dedicated operator
console. These proprietary items were specific to thesystem concerned. As Unix and X-Windows have become
more widespread the systems described in this paper allbase the high level operator interface on these standards,
although more specific interfaces for specialists may befound at lower levels. A final step, described below, is theintegration of high level functionality from several of thesesystems into a single high level application. We believethat a seamless integration of a variety of these systems
will be very important for the efficient operation of thebeams in the LHC.
A homogeneous Application Program Interface, (API),to equipment remains an elusive goal at CERN. The two
accelerator control groups have evolved different standards. The PS group [6] has a strict syntax which contains
elements essential for the Pulse to Pulse Modulation of the
CPS Complex.
For SPS and LEP the API, SL-EQUIP, evolved rather
late [7]. Emphasis was on compatibility with lower layer
calling protocols and ease of usage in order to encourage
rapid acceptance. These aims have proved incompatible
with the ideal of defining a common standard for all accelerators. The SL-EQUIP package hides details of the Control System from the client application program. All callsfrom application software follow the Master-Slave model.

SL-ALARM is a complementary package to SL-EQUIP
which defines a simple API to the CERN Alarm System.
Communication between industrial systems and external
software for the operation of the SPS and LEP accelerators
is based on the SL-EQUIP and SL-ALARM packages. To
date the PS Complex has not addressed this issue.
More recently [1] a third approach has been introduced.
This was driven by the goal of providing a fast, reliable,consistent interface to site services data in the CERN
technical control room. The Technical Data Server (TDS)
maintains a centralized image of equipment status that is
updated on change of the monitored values. Built on the
Talarian middleware package RT-Works all data entering
the system passes through the generic RT-DAQ module.
CERN has added a layer below this, the Generic TDS
Equipment Access Protocol defining the interface to berespected by all systems. Initial experience implementing
this interface with an existing DCS has been successful.
Existing equipment entering the package uses SL-EQUIP
on the equipment side of this layer. SL-EQUIP does notsupport an event driven approach so values must be polled
here; however, the RT-DAQ module then implements apublish and subscribe approach. The TDS will achieve asingle integrated high level interface to a variety of site
services systems for the Technical Control Room at
CERN.
A common API at CERN would facilitate software
maintenance and provide a unique interface when dealing
with external suppliers. At present a suitable standard toachieve these goals seems a long way off.
3.3 Industrial software integration
Software developed to CERN specifications for use in
the Control System has been fully integrated into thenative software environment [8]. However, the approachwith the more extensive systems described here has been
the opposite. The external supplier has delivered thegateway computer and all system and application software.
The intent is to establish clear responsibilities; the software
supplied is isolated from the main body of application
software.
We are currently reviewing our software configuration
tools. One of the objectives will be to improve theintegration of high-level gateways. A better control of thesoftware environment would permit us to validate external
software in a defined environment allowing CERN to
supply the host computer and system for these interfaces.
In particular, it is important to fully control the interdependencies between industrial systems themselves andin-house software. We have used the SL-EQUIP and SLALARM packages for exporting the functionality of thesesystems upwards. The packages are supplied by CERN to
the vendor who is then responsible for their integration
into his environment. Taking the Machine Access system
as an example the SL-EQUIP software runs in the HP-UX
workstation. The server-side software of SL-EQUIP allows
the Experimental Teams to give access from their generalbeam-line control programs. While all SL-EQUIP servers

can be traced through the Equipment Database a means oftracing their clients is missing in this system.

running on that platform at least during the guarantee
period.

3.4 Global alarm retrieval

4 Conclusions

The CERN alarm system federates data from a wide
variety of sources connected to the CERN Accelerator andServices Network. This facility has developed from the
LEP Alarm System [9] which was built to manage thealarm messages arising from the LEP machine and thetechnical services. The role of this system has since been
extended to the Technical Services for the whole of the
CERN Site and the SPS Machine. Alarms are reported by awide variety of systems and routed to a central serverwhich sorts and dispatches them to various control rooms
on the CERN site. A very important feature of the system
is to be able to present, for example, the Technical
Services operator with a single interface allowing him tosee alarms from the various technical services at CERN
such as electricity, water, ventilation, fire detection. The
system can also provide a consistent chronology ofimportant events involving several systems, cooling pumps
losing their electrical supply is a simple example. During
the early phases of this development, systems were
integrated through the standard LEP architecture [10]connecting extensive DCS systems into the control systemthrough a protocol converter attached to the MIL 1553
multi-drop bus. More recent systems inject alarms throughthe SL-ALARM package running in a UNIX workstation
connected to the Machine and Services network. The
workstation becomes the protocol converter and offers the
manufacturer a powerful and familiar environment, which
can also be used for other system components such as
supervisory systems.

For several years CERN has made a large effort tointerface and to integrate industrial systems in its accelerators and services controls. Today all general technicalservices are industrial, either as complete turnkey systems
or as individual clusters of PLCs. The integration practices
are still evolving with the industrial offer; but TCP/IP andstandard fieldbuses are becoming widely supported byindustry. The new trend is to extend the use of industrial
systems to pure accelerator controls for which dedicatedequipment has always been designed and built in the past.
Our future goal remains the smooth integration ofindustrial systems for the LHC operator.

3.5 Specifications and project management [11]To achieve a clean and well-managed integration ofcomplex systems it is essential to define interfaces andresponsibilities. For systems where the CERN network isused to communicate between components of theindustrial system CERN supplies the communication
service in terms of availability and bandwidth. In return theuser, in this case the system vendor, must abide by certain
restrictions. These are :1) He must agree to the bandwidth consumed by his
system;2) He must only attach stations that can be managed by
SNMP;3) He must use the TCP/IP protocol or his own after prioragreement with CERN;
4) He must not make broadcasts.
For systems where the manufacturer inputs alarms to theControl System or where he exchanges services with othercontrol systems then CERN provides the SL-EQUIP, SLALARM or RT-DAQ software. CERN remains
responsible for this code while it is the manufacturer′s task
to interface his system. CERN prefers the manufacturer tosupply the UNIX platform as part of his turnkey system
and remain responsible for all hardware and software

Acknowledgements
The authors would like to thank their collaborators fortheir help in the preparation of this paper. They are also
very grateful to the control experts in all CERN equipmentgroups for their collaboration in these various industrial
controls projects.
References[1] P. Ninin, et al., Industrial Control Systems at CERN,ICALEPCS′97, Beijing, China, Nov. 1997.
[2] P. Charrue, et al., The Equipment Access Software
for a Distributed UNIX-Based Accelerator Control
System, ICALEPCS′93, Berlin, Germany, Oct. 1993.
[3] M. Tyrrell, The LEP Alarm System, ICALEPCS′91,
Tsukuba, Japan, Nov. 1991.
[4] P. Liénard, et al., A new LAN Concept for LEP
Machine Networks, a Step towards LHC, ICALEPCS'95, Chicago, USA, 1995.
[5] G. Baribaud, et al., Recommendations for the use of
Fieldbuses at CERN, ICALEPCS'97, Beijing, China.
[6] P. Antosanti, et al., Workstations as Consoles for the
CERN-PS Complex, Setting-up the Environ-ment,
ICALEPCS'91, Tsukuba, Japan, Nov. 1991.
[7] N. Trofimov, et al., The Equipment Access Software
for a Distributed UNIX-based Accelerator Control
System, ICALEPCS'93, Berlin, Germany, Oct. 1993.
[8] B. Denis, ICALEPCS'95, Chicago, USA 1995. p244247.
[9] M. W. TyrrelI, The LEP Alarm System, ICALEPCS'91, Tsukuba, Japan, Nov. 1991.
[10] P.G. Innocenti, The LEP Control System, ICALEPCS'89, Vancouver, Canada, 1989.
[11] M. Vanden Eynden, P. Ninin, Project Management as
a Breakthrough at CERN, ICALEPCS'97, this Conference, Beijing, China, Nov. 1997.