Technical Data Server: A Large Scale Supervision System
P. Ninin, R. Bartolome, B. Cantau, C. Delgado, H. Laeger, A. Lund, R. Martini, P. Sollander
Pierre.Ninin@cern.chEuropean Organisation for Nuclear Research - CERN
1211 - Geneva - 23, Switzerland
Abstract
CERN’s Technical Control Room (TCR), is responsible
for the overall surveillance and control of its technicalinfrastructure covering systems such as cooling, airconditioning, electric power distribution, control and safetysystems. In order to prepare this supervision for the era ofCERN ’ sfuture accelerator, the Large Hadron Collider(LHC), we have developed, extensively tested and installed
the Technical Data Server (TDS): an event-driven,
distributed, real time information system which caninterface to different types of equipment data.
To achieve our objectives of an efficient andmaintainable supervisory system with very limited humanand financial resources, we based our development on three
concepts: rigorous application of software engineering
standards, use of a commercially available middleware
package and integration of various industrial control
systems via a commonly accepted communication
standard.
The project adhered fully to the PSS-05 Software
Engineering Standards and its life-cycle approach asconceived and published by the European Space Agency(ESA), though some of the technical and managerial
documents had to be tailored to the particular context of a
CERN in-house project. Throughout the development wefocused specifically on quality factors such as: reliability,testing, programming standards and configuration
management. A dedicated simulator was built and the TDS
underwent systematic tests which covered the maximum
load that it is required to handle.
The TDS has been developed on top of the commercial
middleware package RTworks. This product is speciallydesigned for the building of large scale applications,offering high speed inter-process communication,scalability, reliability and fault tolerance.
With the TDS we have successfully implemented a new
type of industrial system integration at the TCP/IP level.
The generic TDS equipment access protocol (GTEAP)
fully defines the asynchronous data exchange with theequipment level, providing a clear partition ofresponsibilities between the global technical supervisionand the local process controls.
The first technical system and the large scale tests hasshown the TDS to behave as a reliable and robust
supervisory system. A further seven new systems
representing 10,000 tags are being integrated and will
be ready for December 1997.
1 Introduction

At CERN, the Technical Control Room (TCR) monitors
data coming from the electrical distribution, cooling water,
air conditioning, vacuum, cryogenics, safety and othersystems.
In this context, a Technical Data Server (TDS) has been
defined and implemented. It provides data collected fromequipment using a standard TCP/IP interface to the highlevel control software such as Human Computer Interfaces(HCI), a Data Logging System (DLS) and a Central Alarm
Server (CAS). It performs real time data storage anddistribution, alarm filtering, data archiving and playback,and command management in a distributed, multi-platformenvironment. Technical infrastructure data are used byother CERN control rooms and by those responsible for
equipment. The TDS is now operational and several
thousand tags are under integration.
The intention of this paper is to summarise the concepts
and performance of the TDS, the experience gained using
RTworks and the applicability of the ESA PSS-05 Software
Engineering Standards at CERN.
2 Motivation for the project
The TDS objectives are to provide: a fast and reliablechannel for data diffusion (states, alarms, measurements
and command), alarm filtering, an efficient and uniqueequipment access method, a common application
programming interface and a reduced maintenance effort.
The TDS shall provide the permanent availability and highreliability required by the TCR.[1]The TDS has superseded the traditional Request-Reply
method of acquiring and managing data, where HCI andlogging data are obtained by polling equipment and alarm
data are event-driven, by the modern Publish and Subscribe
paradigm.
3 Volume of work
The development of the TDS represents a six man/year
effort from the user specification up to the delivery of theproduct in operation. Five hundreds pages ofdocumentation and 80 000 lines of code have been
produced.
4 Environment
The environment consists of a three-layer architecture:
the Control Room layer, the Front-End layer and theEquipment Control layer.
The Control Room Layer consists of HP servers (HCI,alarms, logging) and Xterminals.

The Front-End layer consists of standard VME racks
powered by PowerPC processors running LynxOSinterfacing various fieldbuses (MIL1553, GBUS, BITBUS,JBUS), HP servers interfacing Programmable Logic
Controllers (PLC) via the Siemens protocol SINEC H1,and manufacturer specific gateways connectable to thebackbone via TCP/IP[2]. The top two layers communicate
via a 100 Mbit/s FDDI backbone via switching devices.
The Equipment Control layer consists of CERN built
devices and PLCs from various manufacturers.
5 Off-the-shelf product: RTworks
The TDS has been designed on top of a middleware
product: RTworks from Talarian Corporation (USA).
RTworks is a suite of software development tools forbuilding time-critical monitoring and control systemapplications. It consists of separate processes for dataacquisition, data distribution, real-time inferencing, andgraphical user interface building. RTworks is speciallydesigned for building applications where large quantities ofdata must be acquired, analysed, distributed, and displayedin real time. It offers high speed inter-process
communication, scalability, reliability and fault tolerance.
The major RTworks components are:RTserver - information distribution server.
RTie - expert system builder
RThci - graphical user interface builder.
RTdaq - data acquisition interface.
RTarchive - intelligent information archiver.
RTplayback - information playback module.
6 Supervision principle
The Publish and Subscribe paradigm of RTworks fits
very nicely to the event driven model defined for the TDS.
The RTworks processes communicate via a dedicatedmessage server. Applications can run on a single
workstation or can be distributed across multiple
processors in a heterogeneous network.
Equipment data are sent by their control devices using
drivers (e. g. SINEC H1, CERN SL Equip) to an
Equipment Controller (EC) which converts them to theTDS tag format and forwards them to the TDS using theGeneric TDS Equipment Access protocol (GTEAP).
The tags are then stored in the TDS’s local real-time
databases (RTdaq) and are published to the subscribingapplications. The client applications receive the data by
subscribing to particular data sets called datagroups. Alldata arriving in the TDS are archived for 72 hours(allowing post-mortem analysis of incidents). Archive data
can be played back in client applications just like the realtime data. The alarms are sent to the expert system (RTie)
using the RTworks Guaranteed Message Delivery (GMD)
mechanism which ensures that no data is lost even in thecase of network or server unavailability. The expert system
performs alarm deduction, filtering and alarm burst
detection.
A special mechanism informs in real-time the relevantclient applications of any non-availability of an element(RTdaq, EC..) in the data transmission chain. Allcommands sent using the TDS are identified, logged andtheir execution status is checked. The TDS application
programming interface (GTAAL) allows any client
application to exchange data with the TDS. It is based on
the reception of pre-defined messages handled as
application call-backs.
Tag definition and system configuration data are held in areference database providing a totally data driven system.
The TDS supervision module monitors the overall systemand centralises the errors treatment.
An anti-flooding mechanism has been implemented
whereby only tags different from their default values aretransmitted at initialisation thus optimising system
performance.
Mimic Mimic Mimic
Diagrams
Digrams
Diagrams

DLS

User

CAS

Aplication programming interface

Archive/Playback

Availability
Manager

DB int + hciTag
LoaderAlarmFiltering

TDS
Static DB
CommandManagerTDS
Supervisor

Data
Server

DataAcquisition

DataAcquisition

TDS Data
Generator

DataAcquisition

Generic TDS Equipment Access Protocol (GTEAP)
EC
Landis 1

LandisVisonik 1
ThermalControlEC
Landis 2

LandisVisonik 2
ThermalControlEC
Siemens
Sinec H1EC
Siemens
TCP

Fire
SPS Fire
Detection
Detection
Meyrin PS
CoolingCooling
SPS
Water West
Circuits
Ventilation
Zone

Figure 1 - TDS Logical Model
7 Performance
The performance requirement where:- average transmission time EC- to client: 1 s- maximum transmission time: 30 s
- system capacity: 100,000 tags
- reliability: no data lost in transmission
A simulator tool was built in order to test the mostextreme operational conditions expected:- 12 client applications- 6 RTdaq- 42 Equipment Controllers sending 10 tag/s each,- 1 RTserver and- 105.560 tags loaded.
- 2 multi-user HP servers
The following tests results were obtained:- average transmission time EC- to client: 200 ms- maximum transmission time: 10 s
- system capacity: passed- reliability: passed
During a 1 minute avalanche, where 45.000 tags weresent, the transmission time rose to 10s without data loss.
Within 1 minute the system was back to its normaltransmission rate.
The test sessions have demonstrated that the TDS has
met its requirements in terms of reliability (no data loss in
extreme situations), long term stability, nominal andavalanche transmission speed, as well as ability to monitor
the required number of tags. The modularity and scalabilityof the system does not limit the system to the 100.000 tags
initially foreseen.
8 Industrial control integration
With the TDS we have successfully implemented a new
type of industrial system integration at the TCP/IP level.
The generic TDS equipment access protocol (GTEAP)
fully defines the asynchronous data exchange with theequipment level. It is based on TCP/IP socket streams. It
provides a clear partition of responsibilities between theglobal technical supervision and the local process controls.
To ensure the system integrity the equipment data has to bedefined as TDS tags in the TDS reference database(TDrefDB) along with the TDS topology which hosts thisnew interface.
At the TDS level, the RTdaq processes have been linked
with the GTEAP library for protocol handling. The RTdaq
is able to manage the connection with several ECs. We
have experience with ECs on PC/OS2, PowerPC/Lynx OS,HP/HPUX.
When integrating a new system within the TDS, the localcontrol system provider will deliver its EC according to theGTEAP specification and the requirements for monitoringby the CERN/SL network management system [5].
9 Benefits
The large scale tests and the first system migrated show
the following benefits:- improved response time of the human computer interface
for equipment survey (HCI)- data integrity and coherence between all the TDS client
applications: UMMI, DLS, and CAS- robustness and reliability of all the processes- no more data loss, handling of alarm burst- the equipment layer devices are not linked any more to
the client applications- software maintenance process is improved- the TDS being fully documented its maintenance could
be easily outsourced- all new modules will adhere to the message based andevent driven concepts- single interface for industrial system integration- the definition of the overall system (equipment data andTDS configuration) in a reference database ensures itspermanent integrity.

10 Software engineering
In order to achieve the proposed goals, and to limit therisk due to the limited experience of the manpower
resources assigned to the project, we adopted the ESA PSS05 Software Engineering Standard [3]. This standard has
been tailored with the help of the Danish software company
CRI to the CERN context, needs, and resources available
for the project.
The ESA PSS-05 Software Engineering Standard wasused for the technical, managerial and administrative tasks(configuration management and project management).
Writing the technical documentation following the standard
was essential to the successful management of the project.
However, in order for documentation to be of a highstandard and to be kept up to date with the evolution of theproject it should be carried out either by specifically trained
staff or developers with significant experience.
Mechanisms for ensuring the validity of the documentswith respect to the software also need to be adopted.
Nevertheless, the standards enabled the team to focus in
details to all aspects of the software life cycle. We used thewaterfall for development and the V model for verificationand validation.
The managerial and administrative tasks of the ESA PSS05 organise the contractual relations between the customer
and the software provider and impose additional paper
work which is not adequate for internal CERN
development. For the data integration phase we are testing
the Goal Directed Project Management (GDPM) methodfrom Erling S. Anderssen & all [6]. It is found lessadministrative and more efficient in terms of team
motivation and project controls.
11 Present situation
A further seven systems representing 10.000 tags arebeing integrated and will be ready by December 1997
(CERN fire and gas detection, cooling water West zone, PS
cooling water, PS ventilation, computer center iced water,
heating plant Meyrin and Prevessin, co-generation plant,
SPS ventilation).
12 Conclusion
The TDS has been developed using modern software
engineering methods and standards. Its “ publish andsubscribe” paradigm provides excellent performance andscalability for integration in CERN ’ s the distributed
environment of the CERN CERN ’ sdistributed controlenvironment. The choice of RTworks from Talarian seems
highly satisfactory in terms of performance and reliability(mission critical applications). The TDS design effort saves
resources: no programming is required to supervise newequipment which only requires data definition, EC
integration, system configuration and HCIs design. The
maintenance of the overall supervisory system is drasticallysimplified.
Nevertheless a 5 man year effort was required tocomplete the project. We had to interface all existingequipment, control rooms applications and redefine thecontrol system structure.
The project development was a fruitful experience of a
CERN-industry integrated team, were the CERN
participants learned much from industrial pragmatism andmethods.
The first technical systems to be integrated and the large
scale tests have demonstrated the reliability and therobustness of the TDS supervisory system. The integrationof the remaining and forthcoming systems bring otherchallenges, particularly in the data management area. The
next development effort in the supervision ofCERN’S technical services will focus on a new generationof human computer interfaces based on JAVA and fourth
generation web server technology to integrate the object
oriented definition of equipment supported by the TDS.
Reference[1] ‘The Technical Data Server for the control of the 100000 points of the Technical Infrastructure at CERN’, P.
Ninin & all, Icalepcs 95.
[2] ‘Industrial Control Systems at CERN’, P. Ninin, E.
Cennini, P. Sollander, Icalepcs 97.
[3] ‘ Software Engineering Standards’ , ESA Board forSoftware Standardisation and Control, 1994[4] ‘ TDS Interface Control Document’ , April 1996, P.
Ninin & all[5] ‘ Data Communication Infrastructure Available at
CERN for Interconnection of Industrial Control
Systems’ , Robin Lauckner, Patrick Lienard, RaymondRauch, CERN-SL 97-14 CO, March 1997.
[6] ‘ Goal Directed Project Management’ , Erlking S
Andersen, Kristoffer V Grude, Tor Haug, Coopers &Lybrand