Augmented reality aiding collimator exchange at the
LHC

Héctor Martínez, Thomas Fabry, Seppo Laukkanen, Jouni Mattilac, Laurent
Tabourot

► To cite this version:

Héctor Martínez, Thomas Fabry, Seppo Laukkanen, Jouni Mattilac, Laurent Tabourot. Augmented reality aiding collimator exchange at the LHC. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2014, 763, pp.354-363. 10.1016/j.nima.2014.06.037 . hal-01275132

HAL Id: hal-01275132

https://hal.science/hal-01275132v1

Submitted on 15 Feb 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Augmented reality aiding collimator exchange at the LHC

Héctor Martínez a,*, Thomas Fabry b, Seppo Laukkanen a, Jouni Mattila c, Laurent Tabourot d

<figure><img src="image_2.png" /></figure>

CrossMark

## ARTICLE INFO

**Article history:**

Received 19 September 2013

Received in revised form

6 June 2014

Accepted 10 June 2014

Available online 20 June 2014

**Keywords:**

Augmented reality

Collimator exchange

Remote handling

Maintenance

Radioactive equipment

## ABSTRACT

Novel Augmented Reality techniques have the potential to have a large positive impact on the way remote maintenance operations are carried out in hazardous areas, e.g. areas where radiation doses that imply careful planning and optimization of maintenance operations are present. This paper describes an Augmented Reality strategy, system and implementation for aiding the remote collimator exchange in the LHC, currently the world's largest and highest-energy particle accelerator. The proposed system relies on marker detection and multi-modal augmentation in real-time. A database system has been used to ensure flexibility. The system has been tested in a mock-up facility, showing real time performance and great potential for future use in the LHC. The technical-scientific difficulties identified during the development of the system and the proposed solutions described in this paper may help the development of future Augmented Reality systems for remote handling in scientific facilities.

© 2015 CERN for the benefit of the Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

## 1. Introduction

Particle physics is a branch of modern physics that studies the smallest known constituents of matter. Particle physicists try to find out what the Universe is made of and how it works. By studying what happens when fundamental particles collide at high energy levels, physicists learn about the laws of nature.

The study of the basic constituents of matter necessitates large and complex scientific instruments. The instruments used at particle physics laboratories are particle accelerators and detectors. Accelerators boost beams of particles to high energies before they are made to collide with each other or with stationary targets. Detectors observe and record the results of these collisions [1,2]. The largest and highest-energy particle accelerator is the Large Hadron Collider (LHC) at CERN in Geneva.

The circulation and collisions of high energy beams in the accelerators and detectors have an undesirable consequence, namely the radiological activation of some of the components of these facilities [3]. This activation affects differently to the constituents of the accelerators and is more pronounced in some components than in others.

One constituent of a particle accelerator is collimators. Collimators are special devices that mechanically narrow the beam of

particles that is accelerated. In modern accelerators, collimators are highly technological devices that are cooled and can become highly radioactive [4] and must be good absorbers [5], extremely robust and work as precision tools [6,7,5].

Because of their complexity, eventual maintenance and/or replacement operations however have to be foreseen. As the collimators will be among the most radioactive components in the LHC, their maintenance and exchange has to be studied in detail [8], and all possible tools for optimization of the operation have to be considered [9].

The LHC collimators especially if installed in the beam line facing the tunnel wall are objects difficult to reach for maintenance work. During design, this was already kept in mind, and to provide a fast exchange a plug-in system was developed, ensuring an efficient installation and removal of the device. However, additional equipment is installed close to the collimator. This additional equipment is complicating the maintenance proceedings and not facilitating the accessibility. Herefore, Augmented Reality (AR) could become a very important tool.

The AR concept refers to the merging of real images with virtual content in real time. Fig. 1 shows AR with respect to the real environment and to virtual environments, as they can be found in CAD programs. In this Reality-Virtuality continuum, defined in Ref. [10], AR appears as an intermediate state between reality and virtuality.

There are several fields, such as medicine, education, entertainment or tourism, that have taken advantage of AR showing good

<footnote>a SenseTrix, PL 20 FI-00101 Helsinki, Finland</footnote>

<footnote>b European Organization for Nuclear Research, CERN, CH-1211 Genève 23, Switzerland</footnote>

<footnote>c Tampere University of Technology, PO Box 527, FI-33101 Tampere, Finland</footnote>

<footnote>d SYMME, Université de Savoie, Polytech Annecy-Chambéry, 5 chemin de Bellevue, 74944 Annecy le Vieux, France</footnote>

<footnote>* Corresponding author.
E-mail address: hector.martinez@sensetrix.com (H. Martínez).</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 1. Reality–virtuality continuum.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. UML component diagram of the proposed system.</figcaption></figure>

results [11–14]. It has also shown solid results for maintenance of complex technical equipment [15–20]. The majority of those works are oriented to human intervention maintenance, where virtual information is displayed in a suitable fashion for the aid of the maintenance worker. The results provided in these publications are that AR is an easy to use tool for maintenance [17,18] that enhances the task efficiency [17–20] and decreases the accident risk [19]. Moreover, AR has already shown good results in terms of training maintenance workers in hazardous facilities, such as nuclear power plants [21].

The results obtained in Ref. [18] show that AR enhances the task localization in terms of time saving and head and neck movements (minimising head and neck movements could potentially reduce overall musculoskeletal workloads and strain related to head movement during maintenance tasks). They compare their AR prototype to a modified version of the traditional system in use in the facility, which they call as LCD. The mean task localization time for the AR case was 4.9 s against the 9.2 s that it took with the LCD condition. The mean translational head exertions were 0.25 m for AR while 0.68 m for LCD and the mean translational head velocities were 0.05 m/s for AR and 0.08 m/s for LCD.

In Ref. [22], quantitative results of the improvement in efficiency are detailed. In the study, three aspects of the Operations and Maintenance (O&M) procedures with and without the use of AR are compared: the time spent on locating the maintenance target area, the time spent on obtaining sensor-based operation data and the time spent on obtaining equipment-specific maintenance information. Here, an average of 51% of the time spent during a task where workers are located in the target areas is saved, while 8% of the time spent during a task is saved while obtaining sensor-based performance data.

In remote handling maintenance, however, the number of AR based works is more limited. There are some cases, such as in hazardous facilities with highly radioactive areas, where the maintenance has to be done remotely for the safety of the workers. The distance factor has to be taken into consideration and the approaches for AR applications have to be different from human

intervention. Some issues that have to be taken into account are, for example, the use of remote imaging systems instead of built-in cameras and the need for guiding aids for the control of remote devices.

In Ref. [23] some of the possible benefits of the use of AR technology for remote handling in radioactive areas are presented. These benefits range from collision avoidance to recording of the work carried out for later review. All these benefits can be translated into a safer and more efficient maintenance performance.

Ref. [24] presents a case study of AR applied to remote handling in an ITER mock-up scenario. In this work, a template based matching algorithm is used to detect and track the Water Hydraulic Manipulator in the video feed. The accuracy achieved in this work is high and the tracking is done in near real-time. However, the markerless tracking used needs around 0.3 s to detect one object in the scene. If we think of a large number of different devices (like the different types of collimators), this approach cannot be used until the required time for the markerless algorithm is reduced to allow the detection and tracking of several devices in real-time.

The work proposed in Ref. [25] presents a series of experiments that aim to improve the depth perception of teleoperation procedures. The target of their experiments is to enhance teleoperations at ITER by overlaying depth cues to the real view. From the different experiments, the results obtained show that the best performance using virtual cues was obtained by using stereo tracking.

The structure of the remainder of this paper is as follows: Section 2 details the developed system from the architecture to the features and difficulties found. Section 3 discusses the results. In Section 4 the conclusions from this work are discussed and the possible lines for future development are proposed.

## 2. Materials and methods

The system proposed in this paper is intended to be used for remote handling and remote maintenance in the CERN environment.
The developed AR system for remote collimator exchange relies on marker detection and provides multimodal interaction and augmentation capabilities. Markers have been the solution selected for the recognition process to ensure the performance in real-time. A database system has been used to provide flexibility for the development and maintenance of the final application.

## 2.1. System description

Fig. 2 shows the UML component diagram of the system. Every component inside the component diagram corresponds to one module of the system.

The main actor (i.e. the user) is in charge of defining the configuration of the camera to be used for the current session. This configuration is fed to the acquisition module which starts the camera with the provided configuration and begins the acquisition of video frames. The frames are fed to the marker detection module and to the rendering module.

The marker detection module receives the frames and starts the recognition and tracking of the markers visible in the image. This module provides two outputs: the unique identifier of the markers and the position and orientation of the markers in the 3D world. This information is fed to the database module and to the rendering module.

The information about the content (e.g. virtual elements associated with every marker identifier, the path of the files, and the information to display in the menu) is located in the database module. All this information is provided to the multimodal augmentation module and to the menu module, according to the information received from the marker detection module.

The multimodal augmentation module receives the information of the content data and processes the files before the actual rendering. This pre-rendering step is needed as the virtual elements to be rendered come from very different sources (e.g. videos, images, audio files, etc.). Therefore, they have to be read by the system and unified for the final rendering. When the content has been unified, it is fed to the rendering module. The multimodal augmentation module is also in charge of receiving the interaction of the user with the system and react accordingly to this input.

The system also provides the option of displaying a menu over the final augmented view. The menu module communicates with the database module to obtain the required information to display in the menu, including the information related to the current maintenance step and provides layer to be displayed to the rendering module.

Finally, the rendering module receives the inputs of the rest of modules (i.e. the video frames, the tracking information, the

rendering information and the menu layer) and renders the final augmented view that is displayed in the screen.

The great majority of AR applications make use of USB or embedded webcam to acquire the live feed of the target scene. However, AR applications for remote handling maintenance are meant to be used from a separate location than the actual maintenance location, making difficult the use of cameras that are attached to the computer. For this reason, one suitable solution is to use IP cameras that can be accessed from distance. For the prototype proposed in this paper, a calibrated Pan-Tilt-Zoom camera (AXIS PTZ 214) has been used. The camera can be manipulated remotely and the video feed can be acquired by the implemented system and processed for the marker recognition and for the augmentation. In the future, the collimator exchange system is expected to integrate up to four cameras in the new crane with an optical fiber transmission of the video feed. We plan to use one (or more) of those cameras for the final AR system.

Nowadays there are several ways of implementing the recognition of the scene that can be divided into two big groups:

* The first group comprises those systems that use black and white images (fiducial markers) to track their positions and use this info for the final augmentation. Several researchers have made use of this approach with different marker configurations [26-28].

* The second approach (usually called markerless) tracks natural features ranging from planar images to 3D objects. This second approach has also been used in a large number of works [29-31].

Although a markerless approach may look more natural, there are a number of disadvantages compared to fiducial markers approach. Markerless techniques need to detect and track featured points in the input image, which is a slow process compared to fiducial marker detection. Markerless detection also requires more memory use and the previous training of the natural features. For the proposed setting (collimator intervention), a markerless approach may be problematic as there are different collimator configurations with similar appearance. This means that the markerless approach may mismatch the detection of the collimator, showing the maintenance steps for the wrong model. Fiducial marker detection is more robust than markerless techniques in this sense. Due to the previous considerations, a fiducial marker approach has been used in the proposed system.

From all fiducial markers, binary-based markers seem to be the most robust [32]. The marker detection module proposed in this paper has been designed to detect binary-based markers, using ARToolKit [33] as the base library. These markers are represented

<figure><img src="image_5.png" /><figcaption>Fig. 3. Left image shows an example of a binary-based marker. Right image shows the real device to maintain (i.e. the collimator) with the attached markers.</figcaption></figure>
as 3 × 3 matrices of black and white squares (see Fig. 3 left). Each square represents a binary number, creating thus a number that uniquely represents every marker. One advantage of this approach is that the markers require no previous training and that it allows lower processing times as markers do not need to be tested against an increasing number of patterns. Table 1 shows results obtained from Ref. [27], where processing times for fiducial markers and binary-based markers are presented. Thus, by using binary-based markers the system can be scalable as the processing time does not increase significantly when a larger number of markers have to be used. Fig. 3 shows an example of a marker (left) and the collimator and collimator crane with the attached markers (right).

The two main aspects in the development of an AR application are the marker/markerless positioning system (explained in the previous paragraphs) and the rendering system. The rendering system used in this work uses OpenSceneGraph [34] as base library. The multimodal augmentation module is in charge of unifying the Human Computer Interaction (HCI) capabilities of the system. It is in charge of the pre-processing of the virtual content (e.g. 3D models, images, videos, plain text, browsers and voice instructions) and of the user input by means of keyboard, mouse and/or touch screens.

There are around 100 collimators along the LHC tunnel (this number could increase up to 152 in future) with up to eight different configurations [35]. For this reason, the number of different 3D models and other virtual elements may be relatively large. Instead of hardcoding the paths of the virtual elements inside the application as a large number of AR systems do (e.g. [36]), a database system has been used in order to create and maintain the system without code modifications. The database

contains all information required to properly load and display the virtual elements in the augmented scene. Moreover, the steps definition process has also been moved to the database system, so that the different steps can be added, edited or modified without the need of recoding the application. The database system has been designed to contain three levels of the maintenance process: maintenance, job and task or step. Maintenance is the top level and can comprise one or more jobs. Each job can contain one or more tasks (steps) which are made up of one or more virtual elements (see Fig. 4). In the database, every maintenance, job and task has its own name, identifier and description. All the info about the elements to be displayed (file path, size, position with respect to the marker, etc.) is also defined in the database. The database has been used locally, but it can be seamlessly used as distributed database with distributed virtual elements. As a result, the virtual elements can be stored and managed in a centralized system or split into different computers and accessed from any device connected to the same network.

The AR view comprises the video feed from the camera and the virtual elements augmenting it in real time. There are some elements that do not need to be included in the 3D world (i.e. they do not need to be associated with any real object). For those elements, a head-up display (HUD) like menu has been designed and included into the AR view. The HUD is semi-transparent and can be hidden at any time by the operator. The goal is to offer as much information as needed in the way that is helping but not disturbing the operator. Fig. 5 shows an example of the final layout of the HUD menu.

## 2.2. Novel features of the proposed system

The AR system has been designed as a step-by-step guiding tool for the maintenance operator. The maintenance comprises two jobs (collimator removal and collimator installation) and each job has several tasks or steps. When the operator launches the application, he or she decides which job is going to be performed and automatically the first step appears on the screen. Each step is different and may contain one or more virtual elements, as well as text or voice instructions. The virtual elements may be the goals to reach or helping content for the particular step. When the system detects that the step has been accomplished, either by automatic detection or by operator input, the elements of the old step are removed and the new step is displayed. The operator is able to navigate back and forth through the steps by using a keyboard if needed. Table 2 details the steps to follow and the information provided by the application to the operator in the augmented view and in the HUD for the collimator removal job.

The collimator exchange intervention implies the movement of the collimator in an environment where sensitive equipment is surrounding the target position (Fig. 6). For this reason, the intervention has to be performed with great precision, avoiding possible damages to other equipment. In order to cope with this limitation, a path guiding system has been developed for the

<caption>**Table 1**</caption>

Values of processing time (ms) for different cases for fiducial markers and binary-based markers. The values have been obtained from Ref. [27]. The number of markers loaded affects only fiducial markers, as the binary-based marker approach does not load any marker in memory before the actual data processing. Therefore, binary-based markers are affected by the number of visible markers only while fiducial markers are affected by both factors: the number of markers loaded and the number of markers visible. As it can be seen in the table, for a low number of markers loaded and visible, both approaches have similar processing times. However, for larger number of markers visible the processing times of fiducial markers increase rapidly compared to fiducial marker case. It can be also noticed that the processing time increases quickly with the number of markers loaded for the fiducial marker case.

<table><thead><tr><th colspan="2">Conditions</th><th rowspan="2">Fiducial marker<br>Processing time (ms)</th><th rowspan="2">Binary-based marker<br>Processing time (ms)</th></tr><tr><th>Markers loaded</th><th>Markers visible</th></tr></thead><tbody><tr><td rowspan="2">100</td><td>15</td><td>17.5</td><td>15</td></tr><tr><td>30</td><td>40</td><td>37.5</td></tr><tr><td rowspan="2">500</td><td>15</td><td>75</td><td>15</td></tr><tr><td>30</td><td>175</td><td>37.5</td></tr><tr><td rowspan="2">1000</td><td>15</td><td>175</td><td>15</td></tr><tr><td>30</td><td>315</td><td>37.5</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** The database scheme as used for the current implementation. It contains three levels of the maintenance process: maintenance, job and task or step. Each level has a unique identifier as well as a name and a description. The identifier is used for a proper registration of the progress of the maintenance. The name and the description are displayed to the operator in those places where it is needed (e.g. in the starting window, in the HUD, etc.). Every task (step) contains a number of virtual elements (i.e. the content) that are also defined by a unique identifier and a large list of fields defining all the required information.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Application layout. The HUD menu is displayed over the AR view.</figcaption></figure>

<caption>**Table 2**</caption>

Steps and provided information in the augmented scene and in the HUD for collimator removal.

<table><thead><tr><th>Steps</th><th>Augmented info</th><th>HUD</th></tr></thead><tbody><tr><td>Follow the path</td><td>3D model of spreader part<br>Path and 3D keypoints<br>Collimator hook</td><td>Text description</td></tr><tr><td>Reach the keypoint(s)</td><td>3D model of spreader part<br>3D keypoints (one at a time)</td><td>Text description<br>Image of controller</td></tr><tr><td>Approach</td><td>3D model of spreader part<br>Collimator hook<br>3D animation of alignment</td><td>Text description<br>Zoom video</td></tr><tr><td>Remove collimator</td><td>3D model of spreader part<br>Path and 3D keypoints<br>Collimator base</td><td>Text description</td></tr><tr><td>Lift collimator</td><td>3D model of spreader part<br>3D keypoint</td><td>Text description<br>Image of controller</td></tr><tr><td>Reach the keypoint(s)</td><td>3D model of spreader part<br>3D keypoints (one at a time)</td><td>Text description<br>Image of controller</td></tr><tr><td>Approach</td><td>3D model of spreader part<br>Collimator base</td><td>Text description<br>Image of controller</td></tr></tbody></table>

procedure. The AR view shows key points that the collimator has to reach. Those key points are spheres located in the same 3D virtual environment as the collimator is and they draw the virtual path that the collimator has to follow. When the collimator (or some specific part of it) reaches the key point, the intersection between both 3D models (collimator and sphere) is detected. This intersection means that the key point has been reached and then the next key point in the path is displayed in the AR view. If the operator follows the key points, the probability of collision with the rest of the equipment is reduced.

The final targets, such as the plate on the crane where the collimator is placed, are also displayed as 3D models augmenting the real object. The target and the key points are flashing during the guiding process in order to facilitate their location from the operator point of view. An example of the path guiding feature can be seen in Fig. 7.

The controller used to operate the crane has four main controls to allow the operator to move the four degrees of freedom of the crane. Although the operators are thoroughly trained for operating the crane, in some cases it may be difficult for the operator to know which control is the most suitable for the required movement. For this reason, an image showing the most suitable control

to use is also displayed in the right side of the HUD menu during the guiding process (this can be seen in Fig. 5).

There are some steps in the intervention where the operator has to be extremely careful. One of those steps is the alignment of the spreader with the hook of the collimator. The tests for the proposed prototype were carried out using a crane without powered rotation of the spreader as a new crane with powered rotation of the spreader was still being built. As a result, the alignment of the spreader with the hook is a complicated task to perform with the current crane, as the control of the spreader is not as precise as desired. The spreader is connected to the crane through a cable, enabling free rotation of the spreader along the cable axis. Due to this, the latch system in the spreader may sometimes acquire the wrong position, making impossible the alignment with the hook. Fig. 8 shows both configurations, the wrong one (left image) and the right one (right image). It has to be noticed that the latch system will be removed in the new generation of cranes which include the powered rotation of the spreader. Although this step will be no longer needed in future versions of the system, we describe it here as it was a crucial step for the collimator exchange with the current crane and one important bottleneck in the process.

For these kinds of steps, additional visual information may help the operator to perform the task. In the proposed system, several visual aids for the spreader hook alignment have been implemented. The first aid is a video displayed next to the collimator showing a virtual representation of the procedure for the step. In this case, a completely virtual representation has been selected instead of an AR representation because the operator needs to properly see the real object in order to perform the alignment. The second visual aid is a zoom area of the hook displayed in the right side of the HUD menu. As it has been explained before, the operator needs to clearly see the area of interest for the approach of the spreader to the hook. The AR view shows the whole maintenance scene, hence a zoom of the area of interest may help the worker in those cases where the crane movement has to be precise. In the proposed system, the zoom of the same camera used for the AR view has been used. The position of the marker placed over the hook has been used to calculate the region that is segmented from the camera image, which is later zoomed and displayed in the HUD menu. The region segmented from the image has its origin in the main image in (–100, –100) with respect to the screen coordinates of the center of the detected marked and the size of the crop is 100 × 100 pixels. The zoom is a 2 × digital zoom using a bilinear interpolation. However, this can be changed
<figure><img src="image_8.png" /><figcaption>Fig. 6. Transverse section through the tunnel around the collimator, showing the various services [8].</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Instructions showing the path to follow for the collimator removal. This step is displayed after the hook-spreader alignment and before the actual collimator removal. When the operator press the right key, the first key point starts to flash, showing the order to follow.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 8. Left image shows the wrong configuration for the latch system, which makes impossible the alignment between the spreader and the hook. Right image shows the right configuration for the latch system during the alignment.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Fig. 9.** Example of visual aids. In the image, the video showing the animation of the task is displayed beside the collimator. The zoom of the hook area is also displayed in the HUD menu.</figcaption></figure>

and instead of showing the zoom of the same camera, a second camera can be displayed, in those cases where a second camera is enabled and provides a closer view of the area of interest. **Fig. 9** shows and example of both visual aids.

## 2.3. Difficulties and guidelines for AR design in remote handling maintenance

The work presented in this paper focuses on the collimator exchange intervention in the LHC tunnel at CERN. During the development of the AR application for collimator exchange a number of technical-scientific difficulties have been identified. These can be extrapolated to other AR based applications for remote handling in scientific facilities. The analysis and solutions presented in this section may thus guide the development process for future AR applications for remote handling. The following subsections detail the most common problems that may appear during the development of an AR application for remote handling in scientific facilities, and how those problems have been faced in the collimator exchange intervention AR system.

### 2.3.1. Radiation

In the current case study, the AR system is used to aid the work in environments with ionizing radiation. In the case of the LHC tunnel, the radiological conditions are different when the accelerator is in use, or when the accelerator is down for maintenance. Indeed, during maintenance, when the beam is off, the ionizing radiation only comes from activated material. During operation, the equipment close to the beam is, on top of this, subject to stray radiation directly provoked by the accelerated particle beam.

The proposed AR system is based on marker recognition. The markers need to be present in the final setup in all elements relevant for the AR application, i.e., the collimator and the crane. The crane is a removable equipment which means that it is subject to radiation only during the collimator exchange, which is only performed when the accelerator is down for maintenance. The markers that are fixed on the collimator crane are thus not critical with respect to radiation hardness. The markers on the collimators, however, stay in the accelerator tunnel during operation. This means that these markers have to be radiation resistant, have a lifetime that at least equals the collimator lifetime and have to be made from a material that does not become activated itself.

Important hereby is that the colour contrast of the marker stays stable, in order to not make it more difficult or even impossible to detect the markers from the camera image. Conventional sticker markers have an additional limitation, as the glue may degrade with the radiation dose.

In Ref. [37], black and white photogrammetric targets have been built for collimator survey. Due to the high radiation level of up to 4 mSv/h in the beam cleaning insertions in points 3 and 7 of the LHC, where 37 collimators are present, standard alignment measurements are not possible. Therefore, new targets have been built to be used in such circumstances. The targets are made 100% from anodized aluminium, which has been proven to be resistant to high radiation levels, and provides suitable colour contrast for the detection. As the goal of the prototype proposed in this paper is the exchange of collimators, the current plans for our final system are to use anodized aluminium markers for the collimator, while conventional sticker markers can be used for the crane.

Radiation leads also to problems in electronic equipment. Thus, the cameras used for the AR system cannot stay in the tunnel and have to be movable. During the current case study, the camera is fixed in the tunnel for the tests. However, the camera proposed for the final prototype will be attached to the new crane, which means that it will be subjected to radiation only during the collimator exchange. If the radiation dose would be very high, a radiation-hard camera would be needed. However, this is not expected to be needed in this case and, therefore, there should not be major problems in the functioning of the camera due to radiation issues.

### 2.3.2. 3D models

Collimators and other scientific equipment are usually designed using 3D software tools before the final construction. However, those 3D models are usually CAD models which file formats are not always compatible with 3D rendering engines. Thus, the need of a format conversion for the 3D models is frequent in order to use them in the AR application.

The models used for the collimator case were created using CATIA V5. These models have been exported to STEP format, imported into 3ds MAX and exported to a suitable format (i.e. IVE files) for the rendering engine. The models designed in CATIA are usually highly detailed (e.g. the collimator model utilised for the prototype contains 2,106,058 vertices and 2,524,451 polygons) as they are the base for the construction of the real equipment. However, those models are not optimised for real-time rendering.
<table><caption>**Table 3**
Values of vertices and polygons for the original and simplified models of collimator and crane.</caption><thead><tr><th>3D model</th><th></th><th>Original</th><th>Final</th></tr></thead><tbody><tr><td rowspan="2">Collimator</td><td>Vertices</td><td>2,106,058</td><td>290,966</td></tr><tr><td>Polygons</td><td>2,524,451</td><td>251,414</td></tr><tr><td rowspan="2">Crane</td><td>Vertices</td><td>101,557</td><td>14,410</td></tr><tr><td>Polygons</td><td>107,221</td><td>10,649</td></tr></tbody></table>

The models used for rendering in real-time Virtual Reality (VR) applications usually contain tens to hundreds of thousands of polygons. Considering that AR not only renders the models but also needs time for processing the video feed, the requirements are more demanding. Therefore, the available 3D models of the devices exceed the reasonable level of detail to be used in a real-time application.

For this reason, the models have been simplified in 3ds MAX by reducing the number of vertices and polygons before the final export (a reduction rate of around 90% of the original vertices has been used). Table 3 presents the values of the vertices and polygons for collimator and crane models in the original model and in the simplified model. The reason behind the polygon reduction is to reduce the size of the file, which will reduce the use of memory by the application, by losing details in the model while maintaining a useful shape of the model. In maintenance-oriented applications, the high detail of the models is not a crucial issue, as the models are only intended to guide the operator. Therefore, polygon reduction will not affect the perception from the operator.

As it can be seen in Table 3, the simplified models may be in some cases still too large for their use in real-time applications. Therefore, after the polygon reduction, only specific parts of the models are displayed at a time in the prototype. For example, using the 3D model of the hook (565 polygons) instead of the model of the whole collimator (251,414 polygons) reduces considerably the required memory. Also, the loading time is reduced using the specific parts. The average loading time (including database access and loading in memory) of all models when using the specific parts is around 1 s while it takes an average of 2 s when using the full simplified models. Although the difference may not seem to be too large, it may be significant when the number of involved 3D models increases. However, in any case, this process is done only once at the beginning of the execution and it does not affect the framerate.

The main challenge in the model preparation process is to properly align the models with the markers to provide a coherent visualization in the AR scene. The position of the marker in the real environment has to be known beforehand, in order to properly align the 3D object and its pivot point with respect to the real equipment in order to achieve a realistic augmentation.

The process followed to achieve the proper alignment is the following. Accurate measurements of the real position of the center of the marker with respect to a reference point in the device it is attached to (e.g. a corner of the device) are calculated. Later, these measurements are utilised in 3ds MAX to simultaneously align the pivot point of the model with the center of the marker and the origin of the 3D virtual coordinates. With this alignment, the augmented representation is properly aligned with the real device as the units utilised in 3ds MAX are 1:1.

As a conclusion of the issues commented in this subsection, a pipeline involving importing, manipulation and exporting of the models is required before the use in the final application. However, as the majority of the models are already available, the time required for the model preparation is lower than in those cases where the models need to be created from scratch.

### 2.3.3. Path guiding

One important feature in the work presented in this paper is the possibility of guiding the operator through a virtual path made up of virtual key points. The path shows the positions that the collimator has to reach making easier the movement in the equipment-crowded area. However, the position of those key points has to be calculated with respect to real known positions. As it has been explained before, the camera has been manually fixed in the tests for the first prototype. Nevertheless, the position has changed during the different tests, as it has been manually moved several times. Moreover, the camera that will be used in the final prototype will be integrated in the crane, which means that it will be movable and its position will be unknown. For this reason, the reference positions have to be calculated from the objects in the scene and cannot be calculated from the camera location, as it is not static.

In the case of the collimator exchange, the suggested fixed positions are the hook of the collimator and the plate of the crane for the collimator removal and the collimator base for the collimator installation.

### 2.3.4. Illumination and occlusions

The illumination present in the scene may vary from one collimator area to another, as inside the LHC tunnel the light conditions differ from one place to another. For this reason, it is difficult to foresee how the system will react under those conditions. However, the system has been tested against different light conditions inside the mock-up at CERN with good results.

Marker-based AR systems rely in the visual detection of markers in the scene. To achieve the detection, a threshold is applied to a grayscale version of the captured image. The selection of this threshold value depends on the light conditions and therefore it cannot be fixed beforehand. For the proposed prototype, the method for automatic threshold proposed in Ref. [38] has been used. In the tests, the automatic threshold has worked fine and markers are detected under the conditions tested.

It may be the case that at some point of the tunnel the light conditions are too extreme for the automatic threshold to work properly. In these cases, the operator can manually set the threshold used for the marker detection using the keyboard in order to adapt it to the new conditions. With this option, the system will properly detect the markers in the majority of cases, even in poor light conditions.

There can be also cases when the markers are partially hidden or not visible. The reasons may be high reflection over the markers or different objects occluding the view. For those cases, we are currently working in a multimarker setup that will allow the proper positioning of virtual objects if at least one marker of the multimarker configuration is visible. The goal is to integrate this feature in the final prototype.

## 3. Results

The proposed prototype for aiding the remote collimator exchange in the LHC has been tested in a real collimator exchange in the mock-up facilities. The first results have been satisfactory from the performance point of view. The system is able to work in real time, despite the possible bottlenecks, like the IP communication with the camera, the detection of the markers or the display of the virtual content. The prototype has been tested with two cameras, a webcam for the first tests and an IP camera for the final tests. The specifications of the PC used for the tests are shown in Fig. 4. The application is able to provide a framerate of 30–35 fps with a resolution of 720 × 576 while displaying around 10,000 polygons (i.e. when using specific parts of the models). However,
<table><caption>**Table 4**
Specifications of the PC.</caption><tr><td>Processor</td><td>AMD Turion II P560 Dual-Core 2.5 GHz</td></tr><tr><td>RAM</td><td>6 GB</td></tr><tr><td>Graphics card</td><td>AMD Radeon HD 6650 M</td></tr><tr><td>Webook</td><td>Logitech HD Pro Webcam C910</td></tr><tr><td>IP Camera</td><td>AXIS PTZ 214</td></tr></table>

if the full simplified models are used, the framerate drops to 8–10 fps with the same resolution and displaying around 175,000 polygons.

There has been no need to change the automatic threshold for the marker detection during the process and all markers were detected when they were visible in the image. Thus, the 3D models were properly aligned with the real devices. However, it has been seen that the error obtained for points that are far from the marker along the camera axis is large, as suggested in Ref. [39]. Although these distant points have not been used in the proposed prototype, solutions for improving the accuracy along camera axis have to be found. One of the foreseen solutions for this problem is to use a multimarker configuration. With the multimarker configuration, the final position of a marker is ponderated by the calculation of the position of multiple markers which may lead to a higher accuracy, as suggested in Ref. [40]. Eq. (1) shows how the final matrix transformation ($T''$) is calculated from the individual calculations of each matrix transformation with respect to a reference marker ($T'_i$) for the N markers visible in the image. The value of $\rho_i$ corresponds to an error rate calculated from variables observable in the image (e.g. diagonals and area of the markers). With this approach, the accuracy of the detection may increase and the system can be able to recover from possible occlusions:

$$T'' = \sum_{i=1}^{N} (\rho_i T'_i) \quad \text{where} \quad \sum_{i=1}^{N} \rho_i = 1. \qquad (1)$$

The path guiding system has worked as expected, detecting the key point intersections in the virtual world. As a result of those intersections, the system has been able to advance through the steps automatically. However, there are still a few steps that need a confirmation from the operator to be classified as performed (e.g. the hook-spreader alignment is not automatically detected by the system due to its complexity and the operator needs to inform the system that it has been successfully performed in order to receive the instructions for the next step).

# 4. Discussion

## 4.1. Conclusions

In this paper, a first prototype for an AR based application used for the aiding of collimator exchange process at LHC has been presented. AR has been previously applied for human intervention maintenance applications in the literature with good results. However, there are only very few examples of AR applications for remote handling maintenance. The proposed prototype uses AR technology to facilitate the collimator exchange from a remote location.

A modular system has been developed to acquire the video feed and process it in order to provide a final multimodal augmentation. Due to the special characteristics of the target equipment, several issues, such as camera or marker selection, have been taken into account during the development. A database system has been developed and integrated in order to provide a unified method for creating and maintaining the application.

The final view that the operator will use has been designed to be as clear as possible in terms of user experience and different options have been included to allow an intuitive customized view. As a result, the operator can decide which features to display and which features to hide from the view.

The system has been designed as a step-by-step process, which means that the system will guide the operator through the different steps of the maintenance intervention. In addition to more traditional AR features, novel features oriented to remote handling operations (e.g. path guiding) have been developed and integrated into the system.

During the development of the system, the discovering of the main difficulties for these kinds of systems, such as radiation dose or 3D model manipulation, has been done. Once those difficulties have been identified, feasible solutions have been designed in order to provide a more robust solution.

The AR prototype has been developed for assisting the remote handling crane operator in the collimator exchange process. The main goals of the first prototype development have been to build the modular system and prepare it for the final prototype that would use the crane with powered rotation of the spreader, as well as finding the most problematic issues and providing solutions to those problems. The developed system may be now the basis not only for the final prototype, but also for future remote handling systems.

## 4.2. Future work

In the presented work a system has been built for the first prototype of AR aiding collimator exchange. The next natural step is to build the final prototype for the new crane with powered rotation of the spreader, attending to its new characteristics.

If the precision of the system can be enhanced, it may allow the reconstruction of the whole 3D environment, which will enable the creation of a virtual view from the rear of the collimator, providing thus a new point of view that cannot be achieved in the LHC due to special restrictions. The virtual view can be displayed in the HUD area, allowing thus to simultaneously provide two views (i.e. the agumented view and the virtual reconstruction) in the same screen. The benefits from this approach may be numerous, as the potential of the AR view can be enhanced by a second point of view that may deal with the limitations of the former view and the restrictions of placing physical cameras in some areas in the proximity of the collimator.

Another important aspect to develop is the implementation of an authoring tool that enables non-programmers to create and maintain the application. As a result of the use of the database system, the application can be modified without the need for recoding. However, a user-friendly application should be created to allow a faster and easier creation or modification of the application, as new collimator models or different instructions may appear.

As it has been mentioned before, the zoom feature is implemented from the main camera. However, in the future it may be suitable to replace the zoom with the video feed from a second camera that provides a new angle or a closer view.

As the number of different collimators is already large and can be increased in a future, a sustainable system has to be designed in order to enable the increase of new models. The database system is the first step to provide the sustainability. A second step may be the integration of an approach like the one proposed in Ref. [41] where the number of available markers is larger than using the conventional 2D barcode markers.

Another step after the final prototype may be the performance of user tests to assess the usefulness of the system and to acquire new requirements for future development.
Acknowledgements

This research project has been supported by a Marie Curie
Initial Training Network of the European Community's Seventh
Framework Programme under Contract number (PITN-GA-2010-
264336-PURESAFE).

References

[1] Klaus Wille, The Physics of Particle Accelerators: An Introduction, Oxford University Press, 2001.

[2] Steve Myers, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences: Mathematical 370 (1973) (2012) 3887.

[3] Helmut Vincke, Chris Theis, Stefan Roesler, Radiation Protection Dosimetry 146 (4) (2011) 434.

[4] M. Brugger, S. Roesler, Radiation Protection Dosimetry 115 (December 1–4) (2005) 470.

[5] R. Assmann, O. Aberle, G. Arduini, A. Bertarelli, H. Braun, M. Brugger, H. Burkhardt, S. Calatroni, F. Caspers, E. Chiaveri, A. Dallochio, B. Dehning, A. Ferrari, M. Gasior, A. Grudiev, E.B. Holzer, J.B. Jeanneret, J.M. Jimenez, Y. Kadi, R. Losito, M. Magistris, A. Masi, M. Mayer, E. Metral, R. Perret, C. Rathjen, S. Redaelli, G. Robert-Demolaize, S. Roesler, M. Santana, D. Schulte, P. Sievers, K. Tsoulou, H. Vincke, V. Vlachoudis, J. Wenninger, I. Baishev, I. Kurochkin, G. Spiezia, LHC collimation: design and results from prototyping and beam tests, in: Proceedings of the 2005 Particle Accelerator Conference, PAC 2005, pp. 1078–1080.

[6] Alessandro Bertarelli, Oliver Aberle, Ralph Assmann, Enrico Chiaveri, Tadeusz Kurtyka, Manfred Mayer, Roger Perret, Peter Sievers, The mechanical design for the LHC collimators, in: Proceedings of EPAC 2004, 2004.

[7] A. Bertarelli, O. Aberle, R.W. Assmann, S. Calatroni, A. Dallochio, T. Kurtyka, M. Mayer, R. Perret, S. Redaelli, G. Robert-Demolaize, Mechanical Design for Robustness of the LHC Collimators, in: Proceedings of the 2005 Particle Accelerator Conference, PAC 2005, pp. 913–915.

[8] M. Brugger, D. Forkel-Wirth, S. Roesler, Summary of individual and collective doses for interventions at the LHC betatron cleaning insertion (IR7), CERN Technical Note CERN-SC-2005-093-RP-TN, CERN, September 2006. EDMS Document Number: 689173.

[9] Thomas Fabry, Liesbeth Vanherpe, Mathieu Baudin, Chris Theis, Christian Braesch, Bruno Feral, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 708 (2013) 32.

[10] P. Milgram, F. Kishino, IEICE Transactions on Information and Systems E series D 77 (1994) 1321.

[11] S. Lee, J. Lee, A. Lee, N. Park, S. Song, A. Seo, H. Lee, J. I. Kim, K. Eom, The Veterinary Journal (2012).

[12] Héctor Martínez, Rafael Del Hoyo, Luis Miguel Sanagustín, Isabelle Hupont, David Abadía-Gallego, Carlos Sagüés, Augmented reality based intelligent interactive e-learning platform, in: ICAART, vol. 1, 2011, pp. 343–348.

[13] M.R. Mine, J. van Baar, A. Grundhofer, D. Rose, B. Yang, Computer 45 (7) (2012) 32.

[14] Jens Keil, Michael Zollner, Mario Becker, Folker Wientapper, Timo Engelke, Harald Wuest, The house of Olbrich-An augmented reality tour through architectural history, in: 2011 IEEE International Symposium on Mixed and Augmented Reality-Arts, Media, and Humanities (ISMAR-AMH), 2011, pp. 15–18.

[15] M. Hincapie, A. Caponio, H. Rios, E. Gonzalez Mendivil, An introduction to augmented reality with applications in aeronautical maintenance, in: 2011 13th International Conference on Transparent Optical Networks (ICTON), 2011, pp. 1–4.

[16] U. Neumann, A. Majoros, Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance, in: Proceedings of the IEEE 1998 Virtual Reality Annual International Symposium, 1998, pp. 4–11.

[17] F. De Crescenzio, M. Fantini, F. Persiani, L. Di Stefano, P. Azzari, S. Salti, IEEE Computer Graphics and Applications 31 (1) (2011) 96.

[18] S. Henderson, S. Feiner, IEEE Transactions on Visualization and Computer Graphics (99) (2010) 1-1.

[19] S. Benbelkacem, N. Zenati-Henda, M. Belhocine, A. Bellarbi, M. Tadjine, S. Malek, Augmented reality platform for solar systems maintenance assistance, in: Proceeding of the International Symposium on Environment Friendly Energies in Electrical Applications, EFEEA, 2010.

[20] S.J. Henderson, S.K. Feiner, Augmented reality in the psychomotor phase of a procedural task, in: Tenth IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2011, 2011, pp. 191–200.

[21] Ho Bin Yim, Poong Hyun Seong, Nuclear Engineering and Design 240 (December 12) (2010) 4096.

[22] S. Lee, Akin Ö, Automation in Construction 20 (4) (2011) 338.

[23] Ryan King, David Hamilton, Fusion Engineering and Design 84 (2) (2009) 1055.

[24] Z. Ziaei, A. Hahto, J. Mattila, M. Siuko, L. Semeraro, Fusion Engineering and Design 86 (9) (2011) 2033.

[25] C.J.M. Heemskerk, P.T. Eendebak, G.Y.R. Schropp, H.V. Hermes, B.S. Q. Elzendoorn, A.J. Magielsen, Fusion Engineering and Design 88 (October 9–10) (2013) 1969.

[26] Jun Rekimoto, Matrix: a realtime object identification and registration method for augmented reality, in: Proceedings of the Third Asia Pacific Computer Human Interaction, 1998, pp. 63–68.

[27] Mark Fiala, Artag Revision 1, a Fiducial Marker System using Digital Techniques, National Research Council Publication 47419/ERB-1117 2004.

[28] Xiang Zhang, Stephan Fronz, Nassir Navab, Visual marker detection and decoding in AR systems: a comparative study, in: Proceedings of the First International Symposium on Mixed and Augmented Reality, 2002, p. 97.

[29] Hirokazu Kato, Keihachiro Tachibana, Mark Billinghurst, Michael Grafe, A registration method based on texture tracking using artoolkit, in: 2003 IEEE International Augmented Reality Toolkit Workshop Augmented Reality Toolkit Workshop, 2003, pp. 77–85.

[30] Andrew I. Comport, Éric Marchand, François Chaumette, A real-time tracker for markerless augmented reality, in: Proceedings of the Second IEEE/ACM International Symposium on Mixed and Augmented Reality, 2003, pp. 36.

[31] Gerhard Reitmayr, Tom W. Drummond, Going out: robust model-based tracking for outdoor augmented reality, in: IEEE/ACM International Symposium on Mixed and Augmented Reality, 2006. ISMAR 2006, 2006, pp. 109–118.

[32] A. MarkFiala, IEEE Transactions on Pattern Analysis and Machine Intelligence 32 (7) (2010) 1317.

[33] H. Kato, M. Billinghurst, Marker tracking and HMD calibration for a video-based augmented reality conferencing system, in: Proceedings of the Second IEEE and ACM International Workshop on Augmented Reality, 1999 (IWAR '99), 1999, pp. 85–94.

[34] Robert Osfield, Don Burns, Openscenegraph, (http://www.openscenegraph.org), 2004.

[35] R. Assmann, O. Aberle, G. Bellodi, A. Bertarelli, C. Bracco, H. Braun, M. Brugger, S. Calatroni, R. Chamizo, A. Dallochio, The final collimation system for the LHC, in: Proceedings of EPAC 2006, 2006.

[36] Steven Dow, Manish Mehta, Annie Lausier, Blair MacIntyre, Micheal Mateas, Initial lessons from AR façade, an interactive augmented reality drama, in: Proceedings of the 2006 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, 2006, p. 28.

[37] A. Behrens, P. Bestmann, C. Charrondiere, T. Feniet, J.L. Grenard, D. Mergelkuhl, The Collimator Survey Train, 2010.

[38] Nobuyuki Otsu, Automatica 11 (285–296) (1975) 23.

[39] Pierre Malbezin, Wayne Piekarski, Bruce H. Thomas, Measuring ARTootKit accuracy in long distance tracking experiments, in: The First IEEE International Augmented Reality Toolkit Workshop, 2002.

[40] Jong-Hyun Yoon, Jong-Seung Park, Chungkyue Kim, Increasing camera pose estimation accuracy using multiple markers, in: Advances in Artificial Reality and Tele-Existence, Springer, 2006, pp. 239–248.

[41] Héctor Martínez Seppo Laukkanen, Jouni Mattila, A new hybrid approach for augmented reality maintenance in scientific facilities, International Journal of Advanced Robotic Systems, doi: http://dx.doi.org/10.5772/56845 2013 (Man-uel Ferre, Jouni Mattila, Bruno Siciliano, Pierre Bonnal (Ed.), ISBN: 1729-8806, InTech).