<figure><img src="image_1.png" /></figure>

HAL
open science

# Geometric Coherence of a Digital Twin: A Discussion

Abdelhadi Lammini, Romain Pinquié, Gilles Foucault, Frédéric Noël

<figure><img src="image_2.png" /></figure>

## To cite this version:

Abdelhadi Lammini, Romain Pinquié, Gilles Foucault, Frédéric Noël. Geometric Coherence of a Digital Twin: A Discussion. Product Lifecycle Management. PLM in Transition Times: The Place of Humans and Transformative Technologies, 667, Springer Nature Switzerland, pp.227-236, 2023, IFIP Advances in Information and Communication Technology, 10.1007/978-3-031-25182-5_23. hal-03999846

HAL Id: hal-03999846

https://hal.science/hal-03999846

Submitted on 23 May 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Geometric Coherence of a Digital Twin: A Discussion

Abdelhadi LAMMINI¹[0000-0002-6894-6762], Romain Pinquié¹[0000-0001-5913-4714], Gilles Foucault¹[0000-0003-3009-0767] and Frédéric Noël¹[0000-0003-2466-90932]

¹ Univ.Grenoble Alpes, CNRS, Grenoble INP, G-SCOP, 38000 Grenoble, France

abdelhadi.lammini@grenoble-inp.fr

**Abstract.** The topic of the digital twin has been widely investigated recently. However, few works superficially address its geometrical aspect. This paper attempts to clarify what it means to maintain the geometric coherence of a digital twin based on a taxonomy of geometric modifications. Each type of geometric change and existing capture solutions are reviewed. To conclude, the review encourages us to adopt an agile human-centric approach supported by extended reality technologies as a complement to automated supervision processes.

**Keywords:** Digital twin, Geometry, Coherence, Geometric change, Tracking.

## 1 Introduction

The digital twin concept was introduced as the ideal Product Lifecycle Management (PLM) [1]. A digital twin is strongly intertwined with a real twin in a cyber-physical system. If the digital twin is misaligned with the physical twin, then the cyber-physical system is incapable of providing the services expected by the stakeholders. Although there is a bewildering array of studies related to the alignment of the behavioral properties between the digital and physical twins, proposals for maintaining the coherence of structural properties are virtually absent. For instance, the interface between the physical and digital twins in [2] uses the Industrial Internet of Things (IIoT) to update the behavioral properties without considering geometric changes.

This paper identifies the concept of geometric coherency of a cyber-physical system whose properties emerge from the intertwinement of a physical twin and digital twin. The first section concentrates on the definition of the concepts of the digital twin, digital twin coherence, and geometric coherence based on a literature review. The third section introduces a taxonomy of geometric modifications (GM) to be captured to maintain geometric coherency. The taxonomy supports the definition of each GM, the review of existing solutions to capture GM, and their limits according to some PLM phases.

## 2 Literature review

The concept of the digital twin is much more influenced by ever-evolving technologies and business incentives than rational scientific knowledge. From an academic
perspective, some definitions focus on the multi-physics and multi-scale characteristics [3], whereas others concentrate on the level of data integration and the real-time constraint [4]. The analysis of 23 papers on digital twins identified through a systematic literature review since 2010 on Google Scholar led us to a list of 10 key features. Figure 1 illustrates the number of papers containing each feature. A majority of papers consider a digital as an as-built representation of a real system and both are bi-directionally synchronized in near real-time. However, results show that only 13% of studied papers claimed the importance of geometry coherence as a key feature that will be discussed in this paper.

<figure><img src="image_3.png" /><figcaption>Figure 1 Key features of 23 digital twin definitions since 2010</figcaption></figure>

## 2.1 Digital twin coherence

Although the synchronization between the digital and physical twins is crucial, maintaining their coherence is more about finding the right level of fidelity for the pursued goals. However, studies ignore the geometric coherence and focus only on the behavior by developing data-based digital twins for predictive maintenance [5] and manufacturing [6]. The problem with data-based digital twins is that the geometry is either overlooked or considered perfect whereas dynamic behavioral properties' fidelity depends on it. Without capturing geometric changes, one cannot pretend to have a high-fidelity digital twin. For instance, by evaluating three scenarios, [5] shows that missing teeth in mechanical gears impact the wheel speed.

## 2.2 Digital twin geometric coherence

Keeping the geometry coherent requires capturing geometric modifications before updating the geometry of the digital twin. To capture a geometric change, various tracking technologies exist (Image recognition, 3D scan, Infrared, Magnetic sensors, etc.). As far as the author is concerned, very few papers propose solutions to track geometric modifications of a digital twin [8] [9]. Scanning and reverse engineering solutions [7] provide a new geometric model, but they require a time-consuming process that does not need rapid synchronization.

In the next section, we propose to clarify the concept of geometric coherence by developing a taxonomy of geometric modifications and a review of tracking solutions.
# 3 Approach and methodology

Despite the increasing research studies on digital twins, we still miss a formal definition of coherence, especially geometric coherence.

## 3.1 Digital twin coherence

Although the coherence of a digital twin is anecdotally discussed, Fig 2 clarifies it by refining the service “Maintain the coherence” into solution-neutral technical functions:

* Acquire data: monitoring of the structural and behavioral properties of the digital and physical twins.

* Detect deviation: processing of measured data to identify a gap between the properties of the digital and physical twins.

* Make a corrective decision: once the deviation is detected and the geometric modification type is known, a corrective decision has to be made accordingly.

* Make a corrective action: once the corrective decision is made, corrective action should be executed by the digital twin, the physical twin, or both.

<figure><img src="image_4.png" /><figcaption>Figure 2 Process for maintaining the coherence of a digital twin</figcaption></figure>

The solutions to implement each function depend on the type of geometric modification and the context (business strategy, constraints, resources...). The next section will introduce a taxonomy of geometric modifications and the existing tracking solutions.

## 3.2 Digital twin geometric coherence

Geometric coherence can be defined as the level of similarity between the geometry of the physical twin and the geometry of the digital twin. Having a high score of similarity requires capturing changes on the real twin and to update the digital twin or *vice-versa*. The proposed taxonomy (Fig 3), which is inspired by the manipulation of a bill-of-materials, classifies the geometric modifications (GM) into three categories: 1) modification of the position and orientation of a part or an assembly (GM1), 2) addition, deletion, or replacement of a part or an assembly (GM2), 3) modification, addition or removal of material on a part (GM3).

<figure><img src="image_5.png" /><figcaption>Figure 3 Taxonomy of geometric modifications</figcaption></figure>
a) Modification of the position and orientation of a part or an assembly

i) *Definition*

The position and orientation are two parameters to locate a rigid body in space, or
relative to its parent. They can be presented either by a transformation matrix, or a
position and quaternion vectors. Maintaining the geometric coherence (Fig 2) when a
position or orientation is modified consists of acquiring the new positions and
orientations of the components that make up the digital and physical twins before
calculating the deviation value and making a corrective decision to execute a corrective
action on the real twin or the digital twin or both.

ii) *Review of existing capture solutions*

Getting an indoor position and orientation tracking system remains a bottleneck.
Existing technologies to capture changes in position and orientation can be classified
into three categories: 1) IIoT, 2) image processing, and 3) 3D scanning.

Concerning IIoT, [8] created a digital twin of a factory. A function of the digital twin is to detect the position of a forklift and assets using Bluetooth sensors. Despite the accuracy of the sensors (1 meter), the time spent looking for the assets is reduced by 99.79% (from 960 minutes to 2 minutes).

Regarding image processing techniques, [9] developed a deep learning algorithm to estimate the orientation of a physical twin based on camera input. The estimated orientation value can serve to update the orientation of the digital twin with an error below 20%. Planned future work will add a depth camera to estimate the position too. Likewise, [10] developed two image-based measurement systems (an image-based distance measurement system and a parallel lines distance measurement system) to locate a mobile robot in an indoor environment by calculating its coordinates. The measurement errors range from 2.24 cm to 12.37 cm which could be improved by using a high-resolution camera.

Finally, [11] developed a non-contact measurement method using HoloLens. The HoloLens depth camera scans the area in 3D, and then the user's gaze and gestures select the measurement points that serve to calculate the distance, the area, and the volume between the points with a level of precision of 1 centimeter.

iii) *Limits of existing capture solutions*

[8] equipped multiple assets with sensors. The size of the assets is an important parameter, as tracking a small object (e.g. screw) can't be achieved here, since the attachment of the sensor to the object cannot be achieved.

In [9] results were accurate, but the authors face three main issues. First, full or partial symmetrical objects impact the pose estimation process. Second, the shape of some objects appears similar even if they are seen from different angles which causes the same impact on the process. Third, the objects with a dark light-absorbing surface engender a high error while estimating the rotation angles.

In [10] results depend on the camera quality as well as the lighting. In addition, the use
of sensors to track the position is relative to the required precision level. For example,
Bluetooth sensors can track assets in a factory with an approximate accuracy of 1 meter [8] and GPS has a tracking accuracy between 20 and 30 cm[22].

Finally, the first version of the HoloLens offers a position calculation method with an error of about 1 centimeter [11], which is more precise than Bluetooth or GPS but it is still not enough in some situations such as the capture of tool wearing during CNC machining. Generally, the accuracy of 3D scans depends on the scanner's nature, however, the highest the precision level is, the longer the processing time.

## b) Addition, removal, or replacement of a part or an assembly

### i) Definition

There are multiple reasons for adding (e.g. adding new functionality to the system), changing (e.g. replacing a degraded component), or removing (e.g. removal of a functionality of the system) a component or an assembly of a physical twin. Consequently, the digital twin has to be updated. In some cases, the change occurs on the digital twin first leading to the update of the physical twin. In both cases, the process for maintaining the geometric coherence (Fig 1) consists in capturing the bill of materials of the physical twin to compare it with the digital twin before making a corrective decision to take corrective action or not for adding, removing, or replacing a part or an assembly in the digital twin.

### ii) Review of existing capture solutions

Recognizing and capturing the physical twin components is one way to control the presence or absence of objects, and it is mainly done by 3D scanning or image processing from video captures. For example, the Yolov5 object detection algorithm recognizes objects based on camera input [8] and mobile laser-scanned point clouds detect, capture and extract road objects [12]. Model-based comparison is one approach to detect 3D changes based on the conversion of the point cloud into digital surface models (DSMs) that are compared to detect deviations. Change detection relies on the classification of objects. For example, [13] detects the change areas by comparing two DSMs coming from multi-temporal LiDAR data using classification to estimate the percentage of pixels that have been converted from ground to building and vegetation.

As part of the deviation detection process, [14] has compared laser scan point-cloud data to CAD data as an inspection of industrial plant systems. Combing the inspection process tool with the recognition and extraction capabilities previously mentioned can provide a system to capture a real object to be compared with the digital twin CAD model to indicate the differences between the digital and physical twins. There also exists a set of common solutions to capture the presence of an object such as LED-based photoelectric sensors that emit light.

As far as the author is concerned, no studies have considered object replacement detection.

### iii) Limits of existing capture solutions

The accuracy and the time process of the model-based comparison approach depend on the scan precision as well as the classification process. Thus, employing scanning
technology to capture objects, then comparing them to a CAD model is a heavy and
time-consuming process. Moreover, when small objects are drowned in a large scene,
the detection and extraction process is challenging [12]. The utilization of LED-based
photoelectric sensors is limited in the detection context, but it can be used to indicate
in near real-time the removal of a fixed component.

c) Modification, addition, or removal of material on a part

i) Definition

A component undergoes several material changes. Some are made on purpose (e.g. additive manufacturing) to adapt the functionality of the component to a specific situation, others are usually undesired and are due to external factors (e.g. corrosion, cracking, creeping, etc). Shape modifications are of three types [14]:

*   **Formative:** *"The desired shape is acquired by application of pressure to a body of raw material, examples: forging, bending, casting, injection molding, the compaction of green bodies in conventional powder metallurgy or ceramic processing, etc."*

    *The desired shape is acquired by application of pressure to a body of raw material, examples: forging, bending, casting, injection molding, the compaction of green bodies in conventional powder metallurgy or ceramic processing, etc.*

*   **Subtractive:** *"The desired shape is acquired by selective removal of material, examples: milling, turning, drilling, EDM, etc."*

    *The desired shape is acquired by selective removal of material examples: milling, turning, drilling, EDM, etc.*

*   **Additive:** *"The desired shape is acquired by successive addition of material"*

    *The desired shape is acquired by successive addition of material*

By analogy, the undesired material changes can be brought into three categories:

*   **Undesired material modification:** the undesired shape is formed by an external effort applied to the component body, resulting in a material modification (e.g. crack and delamination. The shape modification is either plastic or elastic).

*   **Undesired material removal:** The undesired shape is acquired by external efforts, resulting in material removal (e.g. corrosion, erosion).

*   **Undesired material addition:** The undesired shape is acquired by external factors, resulting in the addition of material (e.g. dental calculus).

ii) Review of existing capture solutions

Whatever the type of geometric deformation (formative, subtractive, or additive), 3D
change detection based on a second multi-temporal point cloud approach is the point-
based comparison based [15]. It uses LiDAR data and an adaptative threshold to
highlight a red zone in the point cloud indicating the modified zone. The accuracy of
this zone depends on three factors: registration errors, thresholds, and a neighboring
number of 3D change detection. Experiments for evaluating the influence of each factor
show that the adaptive threshold approach results have higher accuracy (95%) as the
number of neighboring points increases and when the registration error is better than
0.041 meters. The overall performance of change detection is measured by four
variables: correctness, completeness, quality, and F1 measure.

To detect geometric modifications due to formative shaping, one can use a sensor to
capture in 3D the shape of cables after bending in near real-time[23]. Moreover, a
review of 61 deep learning-based crack detection approaches points out that semantic
segmentation-based provides high accuracy (from 79.99% to 96.79%) in terms of crack
detection and localization [16]. Finally, the demonstration of the linear relationship
between crack propagation and the electromagnetic signal coming out of a sensor provides a low-cost solution for crack detection in structures [17]. The sensor was able to track small changes in the structure i.e. cracks widths below 1 millimeter efficiently. To detect geometric modifications due to the removal of material, one can exploit the visual attribute of corrosion: colors with a tracking algorithm based on images, or textures with deep learning algorithms [18]. Rayleigh ultrasonic wavefield by laser probing was used to detect a non-visible drilled hole on a subsurface of aluminum plate [19]. The measurement accuracy is not examined under different circumstances, but the inspected hole of 0.5 mm in diameter and 5 mm in depth was successfully identified. Finally, to capture geometric changes due to the addition of material, deep learning solutions recognize the overlap welding, that is, the bulge of weld metal beyond the root that looks like a circle that extends out of the component body [19]. Vibration tests can evaluate the limescale buildup on the walls of the pipes, the status of the pipe section, and the rate of deposition [20].

This review shows the extensive use of 3D scans, deep and machine learning algorithms as well as non-destructive techniques for material change detection.

### iii) Limits of existing capture solutions

Detecting changes in a point cloud density with a model-based comparison threshold approach has high data accuracy [15] but it is a complex and time-consuming process. Regarding the tracking of geometric changes of cables and beams, the maximum length of the sensor is 115 cm. The machine learning recognition and detection algorithms showed an important accuracy result, but it is limited to the outside visible side of the component, unlike the method such as non-destructive tests which can capture changes inside a body. Moreover, the trained algorithms work mostly for the formative, subtractive, and additive shaping as well as the common undesired material changes where a supervision system is installed, but they are not able to detect an undesired change. In other words, maintaining the geometric coherence of a digital twin requires training the learning algorithms with massive datasets representing all kinds of changes to avoid overfitting.

## 4 Discussion

The literature review points out the lack of interest in maintaining the geometric coherence of a digital twin. We assume that this is due to the difficulties of automatically capturing geometric changes compared to the capture and update of behavioral properties with IIoT technologies.

Tracking the geometric modifications of a system may be fundamental in different PLM phases. For instance, considering the geometric deviations during the maintenance phase would improve the quality of the digital twin's maintenance predictions, as *"In addition to monitoring the deviations between collected data and expected values, a digital twin can interpret collected data from other perspectives. Comparison between digital twin simulated data and collected data can help determine the failure mode."*
*Digital twin provides a high-fidelity accurate model and keeps updating through the product lifecycle."*[21]. Having an up-to-date geometric definition of a digital twin also helps to improve the design of multibody system dynamics, especially control algorithms. It is also tremendously important regarding the dismantling phase in critical fields like the nuclear energy industry or aerospace to have a clear understanding of the latest product configuration that will facilitate the recycling process.

A complex system can contain so many components (e.g. sensors, actuators, gears, etc.) that tracking all geometric changes is important but idealistic. IIoT can track the positions and orientations of instrumented objects at different scales (factory, machine, etc.) as well as some specific deformations such as crack and wear detection. However, having an up-to-date digital twin means setting up sensors on all parts or at least knowing in advance all the critical parts to monitor. Sometimes components are not easily accessible, reaching them requires deep investigation inside the system, which makes image capturing and scanning difficult. Image processing techniques also rely upon a supervision system to integrate within an operational environment which can be inappropriate. Furthermore, the trained algorithm may potentially fail to recognize uncommon geometric changes (e.g. buckling) because of the insufficient volume of training and testing data to cover all potential scenarios. Regarding the 3D scan solutions, the capture of geometric deformations is highly accurate but the reverse engineering process is cumbersome.

<figure><img src="image_6.png" /><figcaption>Figure 4 Human-In-the-Loop maintenance of geometric coherence of a digital twin</figcaption></figure>

The automatic supervision and update of geometric changes are still, and will always remain, partial. This observation encourages us to promote an agile human-centric approach supported by extended reality technologies as a complement to automated supervision processes. The human ability to detect and communicate common and uncommon deviations remains crucial to maintain an up-to-date digital twin. As a complement to automated supervision processes, the review encourages us to investigate an agile human-in-the-loop approach supported by extended reality technologies that enable us to capture geometric changes and synchronize them with the bill-of-materials stored in a PLM software (Fig 4).
# 5 Conclusion

The main chara of a digital twin is its synarecteristicronization with a physical twin. Studies aiming at maintaining the coherence between both twins concentrate on the behavioral properties using IIoT without considering components' geometry modifications, and components add/removal in the whole product structure. This paper attempts to clarify the concept of geometric coherence with a taxonomy of modifications. A review of geometric deviations capturing technologies shows that existing solutions require heavy automatic supervision systems (full instrumentation with sensors or camera tracking with image processing) which concentrate on intended geometric changes. Finally, we discuss the human role in the process by arguing that augmented reality, especially spatial mapping, synchronized with PLM software could bring a benefit to automatic supervision systems.

# 6 Acknowledgment

This work was supported by the French National Research Agency, grants ANR-20-CE10-0002 (Coherence4D project) and ANR-21-ESRE-0030 (CONTINUUM project).

# References

1. M. Grieves, « Product lifecycle management », Nova Iorque, McGraw-Hill, 2006.

2. N. Julien et E. Martin, « How to characterize a Digital Twin: A Usage-Driven Classification », IFAC-PapersOnLine, vol. 54, no 1, p. 894-899, 2021.

3. P. Aivaliotis, K. Georgoulias, Z. Arkouli, et S. Makris, « Methodology for enabling Digital Twin using advanced physics-based modelling in predictive maintenance », Procedia CIRP, 2019, doi: 10.1016/J.PROCIR.2019.03.072.

4. W. Kritzinger, M. Karner, G. Traar, J. Henjes, et W. Sihn, « Digital Twin in manufacturing: A categorical literature review and classification », IFAC-PapersOnLine, vol. 51, no 11, p. 1016-1022, 2018. https://doi.org/10.1016/j.ifacol.2018.08.474

5. R. Magargle et al., « A simulation-based digital twin for model-driven health monitoring and predictive maintenance of an automotive braking system », in Proceedings of the 12th International Modelica Conference, Prague, Czech Republic, May 15-17, 2017, 2017, no 132, p. 35-46. http://dx.doi.org/10.3384/ecp1713235

6. A. K. Ghosh, A. S. Ullah, et A. Kubo, « Hidden Markov model-based digital twin construction for futuristic manufacturing systems », AI EDAM, vol. 33, no 3, p. 317-331, août 2019, doi: 10.1017/S089006041900012X.

7. M. S. ALNuaimi, E. D. Alkaabi, et A. Ziout, « THE USE OF REVERSE ENGINEERING IN A COMPARISON BETWEEN ENGINEERED AND SOLID HARDWOOD », Proceedings of ICETIT, 2019.

8. K. Vijayakumar, C. Dhanasekaran, R. Pugazhenthi, et S. Sivaganesan, « Digital Twin for factory system simulation », International Journal of Recent Technology and Engineering, vol. 8, no 1, p. 63-68, 2019.
9. T. Sundby, J. M. Graham, A. Rasheed, M. Tabib, et O. San, « Geometric Change Detection Digital Twins », Digital, vol. 1, no 2, p. 111-129, 2021.
https://doi.org/10.3390/digitall020009

10. I.-H. Li, M.-C. Chen, W.-Y. Wang, S.-F. Su, et T.-W. Lai, « Mobile robot self-localization system using single webcam distance measurement technology in indoor environments », Sensors, vol. 14, no 2, p. 2089-2109, 2014. https://doi.org/10.3390/s140202089

11. J. Huang, B. Yang, et J. Chen, « A non-contact measurement method based on HoloLens », International Journal of Performability Engineering, vol. 14, no 1, p. 144, 2018.
doi: 10.23940/ijpe.18.01.p15.144150

12. L. Ma, Y. Li, J. Li, C. Wang, R. Wang, et M. A. Chapman, « Mobile laser scanned point clouds for road object detection and extraction: A review », Remote Sensing, vol. 10, no 10, p. 1531, 2018. https://doi.org/10.3390/rs10101531

13. H. Chaabouni-Chouayakh, T. Krauss, P. d'Angelo, et P. Reinartz, « 3D change detection inside urban areas using different digital surface models », 2010.

14. C. H. P. Nguyen et Y. Choi, « Parametric comparing for local inspection of industrial plants by using as-built model acquired from laser scan data », Computer-Aided Design and Applications, vol. 15, no 2, p. 238-246, 2018.
https://doi.org/10.1080/16864360.2017.1375675

15. D. Liu, D. Li, M. Wang, et Z. Wang, « 3D Change Detection Using Adaptive Thresholds Based on Local Point Cloud Density », ISPRS International Journal of Geo-Information, vol. 10, no 3, p. 127, 2021. https://doi.org/10.3390/ijg10030127

16. Y. Hamishebahar, H. Guan, S. So, et J. Jo, « A Comprehensive Review of Deep Learning Based Crack Detection Approaches », Applied Sciences, vol. 12, no 3, p. 1374, 2022.
https://doi.org/10.3390/app12031374

17. M. Gkantou, M. Muradov, G. S. Kamaris, K. Hashim, W. Atherton, et P. Kot, « Novel electromagnetic sensors embedded in reinforced concrete beams for crack detection », Sensors, vol. 19, no 23, p. 5175, 2019. https://doi.org/10.3390/s19235175

18. A. Matthaiou, G. Papalambrou, et M. S. Samuelides, « Corrosion detection with computer vision and deep learning », in Developments in the Analysis and Design of Marine Structures: Proceedings of the 8th International Conference on Marine Structures (MARSTRUCT 2021, 7-9 June 2021, Trondheim, Norway), 2021, p. 289.

19. K. Imano, « Detection of Drilled Hole on Subsurface of Aluminum Plate with Rayleigh Ultrasonic Wave Field by Laser Probing », Sensors and Materials, vol. 32, no 2, p. 791-797, 2020. https://doi.org/10.18494/SAM.2020.2659

20. D. Kocsis, « Modeling and vibration analysis of lime scale deposition in geothermal pipe», DOI:10.30638/EEMJ.2014.315

21. M. Liu, S. Fang, H. Dong, et C. Xu, « Review of digital twin about concepts, technologies, and industrial applications », Journal of Manufacturing Systems, vol. 58, p. 346-361, 2021.
https://doi.org/10.1016/B978-0-12-823657-4.00010-5

22. Shade, Steven, Madhani, Premal H., "Android GNSS Measurements - Inside the BCM47755," https://doi.org/10.33012/2018.16001

23. https://shapesensing.com/ (accessed 25/03/2002). Visited 01/06/2022