OMG and Distributed Object Database Issues in Detector Construction andQuality Control
W Harris1, J-M Le Goff2, R McClatchey1, N Baker1, Z Kovács1, F Estrella1, A Bazan3, T Le Flour31Dept. Of Computing, University of the West of England, Bristol BS16 1QY, UK
Email: Wayne.Harris@csm.uwe.ac.uk
2
ECP Division, CERN, Geneva, 1211 Switzerland
Email: Jean-Marie.Le.Goff@cern.ch3
LAPP, IN2P3, Annecy-le-Vieux, France
Email: Thierry.Leflour@lapp.in2p3.frAbstract
The CRISTAL project will provide the quality control fora large number of parts to be produced in several countries
for the CMS Detectors. Production will be controlled by
specifying a set of workflows, which, as well as indicating
the production sequence, specify Quality Control processes
for identifying and correcting problems in production.
Thus the system will be able to ensure that all parts aremanufactured correctly and that they meet the standard
required. Production specifications are provided centrally(Central System) but actual production will take place inso-called Local Centres distributed world-wide and loosely
coupled with the central system. The data accumulatedduring manufacture are stored in a local object database
and will be duplicated to the Central System when network
conditions allow. CRISTAL will store all production
history and the measurements permanently, providing freeaccess to physicists and engineers for experiments andmaintenance for many years to come. The complete
process will occur across a number of sites in different
countries, so an Object Oriented Database on the World
Wide Web will be used to synchronise production.
1 Introduction
The CRISTAL system [1] is being developed to monitorand control the production and assembly process of theElectromagnetic Calorimeter (ECAL) for the Compact
Muon Solenoid (CMS) experiment to be run at the Large
Hadron Collider (LHC) facility at CERN from 2005
onwards [2], [3].
The initial phase of the CRISTAL (Cooperating
Repositories and Information System for the Tracking ofAssembly Lifecycles) project is concerned with theproduction and tracking of over 83,000 lead tungstenmono-crystals and their fast electronics to be installed inthe CMS ECAL high energy physics detector. The CMSECAL detector will ultimately be composed of around
1,000,000 parts of about 500 part types. The Crystal
Detector is just one detector in the CMS experiment andthe CMS experiment itself is just one of four to be usedwith the LHC. Each of these parts will have their physicalcharacteristics individually measured and their associatedtechnical documents will need to be stored and managed.

Due to the number of crystals involved in ECAL construction and the very high standard to which each must
be grown there will be a number of Production Centres
located in Russia, China, and the Czech Republic.
Assembly of the crystals with their associated electronics
and mountings will take place in so called Regional
Centres located in Italy, UK, Russia and at CERN which
will also act as the co-ordination centre.
Since the overall costs and timescales of crystal
production must be strictly controlled, the efficiency of theproduction process is paramount. It therefore follows that
quality control must be rigidly enforced at each step in thefabrication process. The CRISTAL system must support
the testing of detector parts, the archival of accumulated
information, controlled access to data, on-line control andmonitoring on all Production and Regional Centres.
This paper describes the specific database communication problems in the CRISTAL system showing thenetwork and database issues that must be addressed andtheir interactions which require a special approach tocommunications within the CRISTAL project. CRISTAL
must be able to interact with many other tools, ensure thatdata will never be lost across the network and it mustensure that production processes can be defined andredefined as necessary. These issues are discussed in moredetail in [4].
2 Detailed requirements of CRISTALAn Engineering Data Management System (EDMS) [5]will be used to manage, store and control all theinformation relevant for the conception, construction andexploitation of the LHC accelerator and experiments during
their whole life cycle estimated to be more than 20 years.
EDMS will define these systems in terms of productbreakdown structures, assembly breakdown structures andmanufacturing work breakdown structures. All theengineering drawings, blueprints, construction procedures,part definitions and part nominal values will be stored inEDMS. Since CRISTAL provides production management
and workflow management facilities to control and trackparts through the manufacturing life cycle up to the final
construction and assembly of the crystal detector it must beable to interact with EDMSs, both now and in the future.

EDMS: Engineering Data Management System for DesignPBSWBS

Product Breakdown Structure

Work Breakdown Structure(Part Specification)(Task Specification)

Mapping from Design to Production
WfMS: CRISTAL Production Management Facility
Production Workflow Definition(Workflow Models)

Production Workflow Instances

Centralised Database for Models

Distributed Databases forProduction Data

(Workflow Enactment)

Figure 1:The EDMS consitutes a centrally available product description and the WfMS consitutes theactual processes required to produce the final product.

The EDMS will involve several components: a ProductBreakdown Strucutre (PBS) (the graph decomposition of
the Product providing the with Part Definition and Part
Compostion relationships); an Assembly Breakdown
Structure (ABS) (the tree generated from the PBS graphwhen navigating from the top holding the Items and theirdecomposition and a Work Breakdown Struture (WBS)(the graph decomposition of the workflow used to create
and calibrate the parts). The relationship between
CRISTAL and an EDMS is shown in Figure 1.
Effectively, CRISTAL will be used to design and build a
distributed information management system to control
crystal production across all centres and so it will become
the central link in a distributed engineering information
management and control system, controlling the production
process of the crystal detector and providing secure accessto calibration and production data.
A workflow system, derived from the PDM, will be usedby CRISTAL to control production in each of the local
centres as well as the shipping of parts between centres.
Since the workflow system will be designed at the central
site, and also since the production process will evolve overthe lifetime of the project, CRISTAL must be able tocommunicate changes to all the local centres via thenetwork communication system and the changed
workflows will be stored on the local databases at each
local centre.
3 Coping with dynamic change
The previous section alluded to the main difficulty facedby the designers of the CRISTAL system: that of dynamicchange to the definitions of the components that it
manages. The crystal detector is still very fluid in its design
but being the inner most detector matters are made worse

because it must start its production process first so that theCMS experiment can be assembled on time. It is envisaged
that particularly in the early stage of production there will
be many changes to the definitions of tasks and parts. If
these definitions are kept statically in the database thenproblems of database dynamic schema evolution will result
when definitions are changed and particularly when objects
migrate from one centre to another.
This will inevitably lead to the problems of schema
evolution in the database, with many objects needing to be
stored under the old schema with long running transactions.
These problems imply that a versioning system as
discussed in [6], [7] and [8] is essential, but also, to copewith future developments of the engineering tools, we haveintroduced the concepts of meta-objects.
Meta-objects are descriptions of the objects which aremanaged by the database. Meta-objects are customisable
without affecting the underlying database schema. The
concept of meta-objects comes directly from the ideas ofreflection [9] that is the ability for a system to manage
information about itself and to change aspects of theimplementation of the system on an object-by-object basis.
The object framework of CRISTAL is a dynamic
structure which holds a description of itself which can be
manipulated and configured whilst the system is running
thus allowing incremental changes to be made. Within thisframework all fundamental object classes are also are metaobjects. In the current prototype these self descriptions
have been done in an ad hoc way without any
thought to descriptions being made available to otherforeign systems which might wish to use CRISTAL
in the future. If CRISTAL is to support a particular
viewpoint such as Calibration, Maintenance or Experiment
Systems Management it requires a corresponding Under-Local
Archive

Local
Archive

Central Storage

Shared Archive
History

Shared Archive
History

Local Storage

Local Storage

SharedDefinitions

Defns

Defns

Defns

Shared Part
Objects andLocations

Local Archive
Partition

Local System Parition

Central System Partition

Local System Parition

Local Archive
Partition

Figure 2: Partitions and data distribution in CRISTAL

lying object schema normalised to that viewpoint in orderto successfully navigate and find relevant data.
4 Data distribution philosophy
The data handling and storage aspects of CRISTAL mustbe transparent to any users of CRISTAL wherever they arelocated. The main objective of the CRISTAL databasesystem should be to provide maximum data availability
with minimum local data storage. Part, task and production
scheme data will be defined by the Coordinator at thecentral site (CERN) and captured in the central database.
The Coordinator will determine when those definitions
become active in the other centres and the CRISTAL
software will distribute these definitions from the central
site to outlying databases; thereafter the CRISTAL
software will populate the new structures in those centres.
CRISTAL must ensure that no data can be lost between
the centres so, a standard distributed transaction model isinappropriate, since a transaction can be blocked or it may
abort due to unreliable or delayed network communications. Special software to handle the hand-over of datafrom one system will be written to ensure that these needs
are met.
Data will be collected at each centre during theproduction/testing/assembly lifecycle and will reside at thecentre for as long as the part resides at that centre. Data
collected locally will be duplicated in the central CRISTAL
system for security, for access from other centres and to act
as input to the final Reference Database. Therefore anobject duplication strategy is required in CRISTAL toprovide transparent access across centres to part definitions
and characteristics. Since the projected final amount of data
in CRISTAL is of the order of 1 TeraByte, a fully
replicated database approach with 10 centres (and therefore

10 TeraBytes of data) is not technically or economicallyfeasible.
Dividing the database into autonomous partitions, eachmanaging and controlling its own data and resources,
allows concurrent access to information across thenetworks and reduces the impact of network failures.
Availability of information, in the presence of failures, isachieved by replicating system resources such as hardware,
software and data. This controlled replication eliminates
any single point of failure in the distributed database. Data
integrity is then maintained by parallel synchronous
updates.
The Objectivity/Fault Tolerance option allows thecreation of a widely distributed database environment. TheCRISTAL database can be divided into autonomous
groups, or partitions, which can be distributed acrossmultiple platforms with access provided by local or wide
area networks. An autonomous partition can continue towork independently of the availability of other autonomous
partitions. Furthermore the Objectivity/Data Replication
option allows replication of the database across multiple
partitions. This replication mechanism again involves
parallel synchronous updates.The CRISTAL databasesystem has been built using Objectivity and has thefollowing set-up. Each local centre is equipped with two
autonomous partitions. One partition holds the local dataand definitions and the other holds the local archive
database (see figure 2) . The central system (at CERN) will
have only one partition for the central database andassociated definitions. The local centres store part
information in the local base relevant for the parts currently
located at that centre. If a part is to be shipped out toanother centre for testing or assembly, the local archive
base and the central database are updated to reflect newinformation and the part information is deleted from thelocal base. The central system’s base always holds all
current part information.

Lieunard, D. Rousset, E. Leonardi, G. Barone, G.
Organtini and W. Harris in creating the CRISTAL
prototypes is also recognised.

5 Conclusions

References:

The above discussion shows the communication needs of
the CRISTAL system. The conclusion of all theserequirements is that the network and database must fit withinteroperability standards, be self describing (and hence
require the storage of the schema and meta-meta-data),
guarantee no blocking or loss of data and store data forextensive periods of time using versioning to adapt tochanging circumstances. OMG and ODMG standards arebeing used for the network objects and databaserespectively, with special versioning, and transaction
models being written.
CRISTAL development was initiated in early 1996 at
CERN and a prototype capture tool was developed using
O2 and WWW to allow existing aspects of ECAL
construction to be incorporated in an object database. The
second phase of prototyping and technology evaluation wasinitiated in the summer of 1996 and development of thisprototype is well under way and is based on the ECAL
testing and construction programme in CMS. TheOODBMS ObjectivityTM is being evaluated as a repository during this second phase of CRISTAL research anddevelopment. At the time of writing the final version of theUser Requirements Document for the CRISTAL Prototype
2 system is complete and the Software Requirements
Document almost complete.

[1] J-M Le Goff et al., "CRISTAL: A data capture andproduction management tool for the assembly andconstruction of the CMS ECAL detector". CMS NOTE
1996/003[2] The Large Hadron Collider Accelerator Project, CERN
AC93-03 1993
[3] M Lebeau, P Lecoq & J-P Vialle, “Distributed Controland Data Base System for the Production of High
Quality Crystals”, CMS TN/95-024, CERN, 1995.
[4] R. McClatchey et al., “bject Databases in a DistributedScientific Workflow Application’. In Proc. 3rd IEEE
Basque Int.Workshop on Information Technology(BIWIT), Biarritz, France July 1997.
[5] A. P. Hameri., "EDMS - Concepts, Motivations andBasic requirements". Proceedings of the CERN Schoolof Computing 96, CERN 96-08.
[6] M. Rusinkiewitz & A. Sheth., "Specification andExecution of Transactional Workflows". In W. Kim,ed., Modern Database Systems: The Object Model,Interoperability and Beyond. Addison-Wesley, 1994
[7] D. Lomet, B. Selzberg, Transaction Time Databases in
Temporal Databeses, ed. Tansel, et al BenjaminCummings[8] W. Cellary, G.Jomier, Consistency of Versions in
Object-Oriented Databeses, Proceedings of the 16th
VLDB Conference, Brisbane, Australia, 1990
[9] J. Peters & M. T. Ozsu, Reflection in a Uniform
Behavioural Object Model, Lecture Notes in Computer
Science 823, R. A. Elmasri et al (Eds) Springer-Verlag,
1994,pages34-4

Acknowledgements
The authors take this opportunity to acknowledge thesupport of their home institutes. In particular, the supportof P Lecoq, J-L Faure and J-P Vialle is greatly appreciated.
The help of A. Bazan, T Le Flour, F. Estrella, S.