# Challenges in Enabling Industrial End-Users to use AR and VR for 3D Authoring of Digital Twins for the Factory of the Future

Clémence Cunin, Thierry Duval, Cédric Fleury

## ► To cite this version:

Clémence Cunin, Thierry Duval, Cédric Fleury. Challenges in Enabling Industrial End-Users to use AR and VR for 3D Authoring of Digital Twins for the Factory of the Future. 4th IEEE VR 2025 Workshop on 3D Reconstruction, Digital Twinning, and Simulation for Virtual Experiences, Mar 2025, Saint-Malo, France. hal-04895024

HAL Id: hal-04895024

https://hal.science/hal-04895024v1

Submitted on 17 Jan 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Challenges in Enabling Industrial End-Users to use AR and VR for 3D Authoring of Digital Twins for the Factory of the Future

Clémence Cunin*

Thierry Duval†

Cédric Fleury‡

IMT Atlantique

Lab-STICC, UMR CNRS 6285 Brest

## ABSTRACT

In this position paper, we present a brief literature overview about the authoring of digital twins in augmented and virtual reality, with a focus on how they can be used by industrial end-users for three common 3D authoring tasks: mesh segmentation, positioning, and behavior creation. This literature overview shows that most tools are lacking some features that are useful and necessary in this context. These features are (i) being cross-platform, (ii) being easy to use by industrial end-users, and (iii) being able to support asynchronous and synchronous collaboration between several users. Last, we detail shortly how our future work will focus on developing a collaborative authoring tool that will enable several industrial end-users to collaborate in an asymmetric way to achieve the three authoring features identified in the literature overview.

**Index Terms:** Human-centered computing → Mixed / augmented reality; Virtual reality; Collaborative interaction; Collaborative content creation; Computer supported cooperative work; Ubiquitous and mobile computing systems and tools

## 1 INTRODUCTION

Digital Twins (DT) offer many opportunities in terms of design, operation, and maintenance of industrial equipment, increasing productivity while maintaining safety in the factory of the future. It can be useful in various situations, ranging from the design of new production lines to remote mentoring of new technicians by experts. The 3D models of these digital twins are not always available, and thus need to be created and authored. Commercial tools such as Blender or Maya can be used to carry out this creation process. However, these tools often require their users to have solid knowledge about how to use them, as well as programming skills. As a consequence, the learning curve for these tools is steep for novices. In addition, employing expert users of such 3D authoring tools can be costly and time-consuming, depending on their access to the industrial site and the need to communicate with the on-site technicians.

In this context, it could be highly beneficial for industrial end-users to be able to directly author or modify a digital twin, as stated by Bégout et al. [2]. In particular, Bégout et al. presented a use-case scenario in which users have to deal with highly reconfigurable production lines, which can be reconfigured several times a day, marginally or more globally. Enabling direct authoring by industrial end-users is thus beneficial for the following reasons: (i) they have a full knowledge of the physical factory, (ii) they can assess the accuracy of the digital twin directly on-site, (iii) they can make small modifications very often to keep the digital twin updated in real time, e.g. every time they reconfigure a production line. All

of this reduces the time and costs associated with the numerous iterations and understanding issues that might be encountered with 3D authoring tool experts. Moreover, many stakeholders can be involved in the authoring process, including operators on production lines, technicians who maintain and reconfigure them, and engineers who design them. As a consequence, collaboration between several industrial end-users can be mandatory to author this digital twin.

3D authoring in an immersive environment has demonstrated benefits as stated by Lee et al. [10], as users can get immediate feedback on their actions in the way of "What You Experience Is What You Get" (WYXIWYG). In particular, Virtual Reality (VR) and Augmented Reality (AR) could support authoring by users who are not experts in 3D authoring tools and collaboration between multiple users. Nevertheless, some software tools still have a mandatory authoring step on desktop computers because of the precision and complex configuration required for creating and modifying 3D models before placing them in the virtual environment. Having authoring tools that can be used simultaneously on various devices or platforms could be advantageous in this industrial context.

This position paper aims to explore the next steps in research to improve the authoring of digital twins in AR and VR for the factory of the future. After a brief literature overview that presents the limitations of the existing tools, we describe what the main research axes in this domain could be.

## 2 RELATED WORK

In this brief literature overview, we include previous work that explored immersive authoring of 3D content in AR and VR. Most of these systems target non-expert users who want to create or modify 3D models without a strong background in 3D authoring tools, computer-aided design (CAD), or programming. We extend this literature overview to other application domains, as only a few previous systems focus on industrial contexts. In this section, we assume that an initial 3D reconstruction has already been completed, and therefore we exclude previous work related to such 3D reconstruction techniques based on depth camera [18] and potentially combined with machine learning-based content generation [9]. We choose to classify the selected previous work into three categories according to the authoring actions performed by end-users: (i) segmenting meshes to extract specific objects from a 3D reconstruction, (ii) positioning virtual objects to complement the digital twin, (iii) defining the behaviors of some specific objects of the digital twin.

### 2.1 Mesh segmentation

A first challenge for authoring digital twins for the industry is that production lines and industrial equipment contain many movable objects or parts that can be manipulated by operators. To address this challenge, some tools enable the user to segment and modify the 3D mesh directly in the virtual scene by adding affordance to it. AffordIt! [11] allows segmentation of the object mesh through direct manipulation, whereby the user can cut through the mesh with a cutter tool to create a rectangular or cuboid shape and thus

<footnote>*e-mail: clemence.cunin@imt-atlantique.fr</footnote>

<footnote>†e-mail: thierry.duval@imt-atlantique.fr</footnote>

<footnote>‡e-mail: cedric.fleury@imt-atlantique.fr</footnote>
create behaviors. Vargas González [14] has compared AffordIt in AR, VR, and with a desktop computer, and concluded that even if the most precise environment for authoring remains the desktop version, no differences were found between immersive and non-immersive methods in terms of defining behavior and visualization preference. It shows the current lack in the field of having an immersive tool that has the same strength as a desktop interface. Cas-sola et al. [3] have developed an authoring tool to improve training in the industrial field, allowing the user to select minor parts of the 3D model to interact with. In addition, other works avoid the ac-cessibility issue of 3D models by scanning real environments with a depth camera and recreating them in VR [9]. Then, the user can in-teract with it by selecting a specific object with a brush in the scene and finding a corresponding existing 3D model object in a database. This is an interesting approach, but unfortunately, it cannot always be used for industrial engines, as 3D models do not always exist already. Finally, other works are working with neural networks to do 3D mesh segmentation [7]. However, it is difficult with this approach to know which parts are interactable and how users will interact with them.

## 2.2 Positioning

A second challenge is that digital twins cannot be always up to date as the production lines can be modulated each day, for ex-ample. To achieve this, some tools for positioning modular parts already exist in the industrial field. ACARS [17] is one of the first context-sensitive authoring systems that gives assistance to main-tenance technicians by displaying guidelines within an AR-HMD. The authoring part of the instructions is part of a desktop compo-nent. This tool can adapt its abstraction level to suit its user's exper-tise level, which is a first step to the development of tools targeting end-users. WAAT [1] and WARD [2] are both developed to dis-play instructions to technicians within AR on a digital twin of the production line. WAAT is composed of a desktop part, to create the 3D scene (with existing 3D object models) and position the models, and another part in AR, to modify and verify the well positioning of the models. WARD adds to WAAT digital twin positioning AR features, making it possible to retrieve existing 3D models from a database and display them. These three tools are for an AR uti-lization with a switch of platform on desktop. Some other works, such as Corsican Twin [13], propose to switch between VR and AR: users can first create visualizations in VR, remotely from the industrial site, and then they can see these visualizations in situ in AR.

## 2.3 Behaviors creation

A third challenge when authoring digital twins is to give an appro-priate behavior to animated objects of production lines or industrial equipment. To define such behaviors, visual programming is of-ten used for creating data-flows, like Ivy [5] and Flowmatic [16]. In these systems, the user can visually create links between com-ponents of objects to add behaviors and events in the scene. This type of programming makes possible fast creation and reduces er-rors by not authorizing the user to establish incoherent connections between two objects. Other tools enable the user to author ani-mation with visual programming, such as the immersive animation authoring tool proposed by Gaarsdal et al. [6]. It enables users to achieve frame-to-frame editing by directly modifying the orienta-tion and size of the objects. Programming by demonstration [4] is another way to enable industrial end-users to create 3D envi-ronments, where they can directly grab objects and define behav-iors. Some other tools, based on WebVR, rely on simplified desk-top interfaces, where the user only has to interact with wizards and components, limiting the effort needed to create a virtual environ-ment [15].

## 3 EMERGING RESEARCH TOPICS

These three different authoring actions demonstrate that different devices can have different benefits according to the task at hand. As seen in section 2.1, the mesh segmentation can be more precise on a desktop computer, whereas it might be more intuitive to perform it with direct manipulation in an immersive environment. As pre-sented in section 2.2, the initial design can be done in desktop mode, especially when achieved by 3D designers, but positioning and ad-justment are easier with AR in situ, especially when they can be achieved by operators. VR can also be useful for method engineers to remotely create the digital twin or to check the final result. As mentioned in section 2.3, the behavior creation seems more under-standable for end users using visual programming or programming by demonstration, thus involving an immersive environment, either in AR or VR. However, visual cluttering can occur for complex sys-tems as the number of objects and behaviors becomes numerous. In this case, it would be better for industrial end-users to be able to di-vide this behavior assignment process into two steps, the first with the desktop, which would be completed with the immersive one.

As shown in these three sections, cross-platform authoring tools can be useful for each authoring action. In particular, these tools can be used by different categories of users to complete a digital twin, at least asynchronously, especially when the interaction is de-signed to address the capabilities of each category of users. Lastly, it should be very interesting to enable these users to also collab-orate synchronously to be more efficient. Indeed, some research papers in the industry of the future domain [18] present collabora-tion between experts and novices, where experts lead novices. This collaborative aspect would deserve more recognition as it makes it possible for different users to merge their different expertise levels during the digital twin authoring.

Some cross-platform tools, making it possible for several cate-gories of users with different roles to collaborate, already exist in other contexts. For example, XRDirector [12] enables different users to embody different roles in a collaborative movie creation context. Each user has a specific task assigned and uses a different device (VR-HMD, Handheld AR) to interact depending on which is the most suitable. In addition, in XRDirector, one user is a director and therefore can take control of every object in the scene, contrary to the other users with limited interaction ability.

Thus, following this analysis, some criteria emerge for develop-ing an authoring tool for digital twins:

* these tools should enable use on different platforms (AR-HMD, VR-HMD, Laptop, Handheld AR) according to spe-cific tasks;

* these tools should support both synchronous and asyn-chronous collaboration;

* these tools should be suitable for different categories of users including industrial end-users by providing easy-to-use and intuitive interaction metaphors.

## 4 DIGITAL TWIN AUTHORING UPCOMING CHALLENGES

We recall here that we do not focus on how the 3D models of digi-tal twins are created, but only on how industrial end-users can seg-ment the meshes, position reconfigurable parts, and add behaviors to digital components. We assume that the digital twin is already at least partially existing, even in a basic form, and that some 3D models can be retrieved from computer vision. So here we will only develop the three main topics identified at the end of the previous section, these are the upcoming challenges that the next generation of digital twin authoring tools must tackle.
## 4.1 Cross-platform ability

Knowing that using a computer or an AR-HMD does not provide the same authoring capability, we believe that a good tool should enable users to do all the basic necessary tasks in terms of digital twin authoring, from mesh segmentation to behavior definition. In order to maintain accessibility and freedom, users should be able to choose between different platforms to author such as AR-HMD, VR-HMD, standard computer, or even handheld AR devices, depending on what they have access to at the moment and what is the most precise and easy to use for a specific task at a specific location. For example, a good authoring tool should be usable with AR when authoring on site and with desktop or VR when authoring from a remote facility building.

## 4.2 Collaboration ability

To our knowledge, existing authoring systems do not support collaboration. So, a good authoring tool should enable all the users, with different levels of expertise, to collaborate in order to exchange on the digital twin at the same time. First, an asynchronous collaboration feature would allow different users to contribute to a common edition by bringing different kinds of expertise. Of course, it could also simply enable a user to switch authoring between several platforms. Second, a synchronous collaboration feature would be useful in a situation where a new production line worker would need authoring instructions from an expert who could only interact remotely. This collaborative feature should include various synchronization features to ensure the consistency of the collaborative virtual environment, as it requires large 3D model data transfer.

## 4.3 Usability for industrial end-users

A good tool should also be suitable for several categories of industrial users such as technicians on the production lines or even engineers who are acting remotely. To make it possible, authoring tools should be very intuitive and context-aware of the different user expertise levels. This is why authoring tools should be suitable for industrial end-users (e.g. operators) with background knowledge neither in programming nor in 3D design.

## 5 FUTURE WORK

The previous brief literature overview and the emerging axes of research gave us some leads to our future work as we are planning to reach a human-in-the-loop approach. For the three parts of 3D authoring including mesh segmentation, mesh positioning, and behavior creation, we can associate our identified criteria.

For the segmentation step, we think that enabling cross-platform interaction within synchronous collaboration would lead to increasing authoring performance as multiple stakeholders could share their expertise and point of view at the same time. This share of knowledge is quite important at the first step of authoring, as it would avoid late error detections. For mesh positioning, the exploration of the use of AR should be our first priority, as in-situ visualization looks promising, but the use of VR seems useful to include remote experts during the authoring. Finally, for behavior creation, the exclusive use of immersive devices, like VR-HMD, seems a bit complicated, as it is not directly intuitive and can induce visual cluttering as the system becomes more complex over time. Here again, a collaborative and cross-platform authoring looks like the best direction for our implementation.

For example, such a collaborative and cross-platform authoring tool would enable a desktop user to collaborate with a VR user or an AR user with different complexities of tasks to achieve. The desktop task can be close to what Unreal's BluePrint proposes, staying in a visual programming approach, while the immersive task would rely on a direct manipulation of the object enabling animation [6].

Several user experiments will be necessary to assess the usability of our prototypes and to characterize the newest interaction

metaphors implemented while maintaining the consistency of the collaborative system. That is why we planned to begin with the evaluation of a mono-user VR version to focus on the interaction parts. Then, we will be able to implement the collaborative part, and finally to implement the AR version. This AR version of the interaction metaphors would be challenging, since they seem to be more limited; this is where direct free-hand manipulation would seem the best option. Bimanual manipulation has already been evaluated as an easy-to-learn technique for novices [8].

For now, we will use a Fischertechnik factory and its digital twin (see Fig. 1) to develop, test, and evaluate our future tool, as a scale model of a factory is more accessible than the real factory.

## 6 CONCLUSION

This position paper aims to identify the main research issues for the creation of digital twins in Augmented and Virtual Reality by industrial end-users. The few current existing tools for the industrial fields lack some features that can be found in other kinds of authoring tools and that we are judging useful and necessary to add in novel authoring tools. The three main features that we have identified for a good authoring tool are: (i) being cross-platform to maintain accessibility, (ii) being able to support synchronous and asynchronous collaboration to help knowledge sharing among stakeholders, and (iii) being easy to use by industrial end-users to benefit from their knowledge of the physical twin.

Our future work will focus on developing a collaborative authoring tool that will enable several industrial end-users to collaborate in an asymmetric way. A typical situation would involve a first user in front of the physical twin using AR, a second user immersed in a Virtual Reality instance of the digital twin, and a third user in front of a desktop workstation working on a task-oriented visualization of the shared digital twin.

<figure><img src="image_2.png" /><figcaption>Figure 1: The real Fischertechnik factory (top) and its digital twin (bottom) that will serve as a prototyping facility for developing and testing our future tool.</figcaption></figure>
## ACKNOWLEDGMENTS

This work was funded by the 5G Metaverse project by BPI France thanks to the PIA4 French research program. This work was also supported by a French government funding managed by the National Research Agency as part of the France 2030 program, reference ANR-22-EXEN-0002 (PEPR eNSEMBLE / CATS). This work was also supported by a French government funding managed by the National Research Agency under the Investments for the Future program (PIA) grant ANR-21-ESRE-0030 (CONTINUUM).

## REFERENCES

[1] P. Bégout, T. Duval, S. Kubicki, B. Charbonnier, and E. Bricard. WAAT: A Workstation AR Authoring Tool for Industry 4.0. *International Conference on Augmented and Virtual Reality*, (12243):304–320, 2020. doi: 10.1007/978-3-030-58468-9_22 2

[2] P. Bégout, S. Kubicki, E. Bricard, and T. Duval. Augmented Reality Authoring of Digital Twins: Design, Implementation and Evaluation in an Industry 4.0 Context. *Frontiers in Virtual Reality*, June 2022. doi: 10.3389/frvir.2022.918685 1, 2

[3] F. Cassola, M. Pinto, D. Mendes, L. Morgado, A. Coelho, and H. Paredes. A Novel Tool for Immersive Authoring of Experiential Learning in Virtual Reality. In *2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)*, pp. 44–49. IEEE, Lisbon, Portugal, Mar. 2021. doi: 10.1109/VRW52623.2021.00014 2

[4] E. Chauvergne, M. Hachet, and A. Prouzeau. Authoring Interactive and Immersive Experiences Using Programming by Demonstration. In *Interaction Homme-Machine*. Troyes, France, Apr. 2023. doi: 10.1145/3583961.3583981 2

[5] B. Ens, F. Anderson, T. Grossman, M. Annett, P. Irani, and G. Fitzmaurice. Ivy: Exploring Spatially Situated Visual Programming for Authoring and Understanding Intelligent Environments. In *GI '17: Proceedings of the 43rd Graphics Interface Conference*, 2017. doi: 10.5555/3141475.3141507 2

[6] J. Gaarsdal, S. Wolff, and C. B. Madsen. Evaluating Immersive Animation Authoring in an Industrial VR Training Context. In *2022 14th International Conference on Signal Processing Systems (ICSPS)*, pp. 809–817. IEEE, Jiangsu, China, Nov. 2022. doi: 10.1109/ICSPS58776. 2022.00147 2, 3

[7] D. George, X. Xie, and G. K. Tam. 3d mesh segmentation via multi-branch 1d convolutional neural networks. *Graphical Models*, 96:1–10, 2018. doi: 10.1016/j.gmod.2018.01.001 2

[8] D. Hayatpur, S. Heo, H. Xia, W. Stuerzlinger, and D. Wigdor. Plane, Ray, and Point: Enabling Precise Spatial Manipulations with Shape Constraints. In *Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology*, pp. 1185–1195. ACM, New Orleans LA USA, Oct. 2019. doi: 10.1145/3332165.3347916 3

[9] A. Ipsita, H. Li, R. Duan, Y. Cao, S. Chidambaram, M. Liu, and K. Raman. VRFromX: From Scanned Reality to Interactive Virtual Experience with Human-in-the-Loop. In *Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems*, pp. 1–7. ACM, Yokohama Japan, May 2021. doi: 10.1145/3411763.3451747 1, 2

[10] G. A. Lee, G. J. Kim, and M. Billinghurst. Immersive authoring: What You eXperience Is What You Get (WYXIWYG). *Communications of The ACM*, 48(7):76–81, July 2005. doi: 10.1145/1070838.1070840 1

[11] S. Masnadi, A. N. V. Gonzalez, B. Williamson, and J. J. LaViola. AffordIt!: A Tool for Authoring Object Component Behavior in VR. In *2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)*, pp. 740–741. IEEE, Atlanta, GA, USA, Mar. 2020. doi: 10.1109/VRW50115.2020.00221 1

[12] M. Nebeling, K. Lewis, Y.-C. Chang, L. Zhu, M. Chung, P. Wang, and J. Nebeling. XRDirector: A Role-Based Collaborative Immersive Authoring System. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems*, pp. 1–12. ACM, Honolulu HI USA, Apr. 2020. doi: 10.1145/3313831.3376637 2

[13] A. Prouzeau, Y. Wang, B. Ens, W. Willett, and T. Dwyer. Corsican Twin: Authoring In Situ Augmented Reality Visualisations in Virtual Reality. In *International Working Conference on Advanced Virtual Interfaces*, p. 11. Salerno, Italy, Sept. 2020. doi: 10.1145/3399715.3399743 2

[14] A. N. Vargas González, B. Williamson, and J. J. LaViola. Authoring Moving Parts of Objects in AR, VR and the Desktop. *Multi-modal Technologies and Interaction*, 7(12):117, Dec. 2023. doi: 10.3390/mti7120117 2

[15] E. Yigitbas, J. Klauke, S. Gottschalk, and G. Engels. VREUD - An End-User Development Tool to Simplify the Creation of Interactive VR Scenes. In *2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VLHCC)*, pp. 1–10. IEEE, St Louis, MO, USA, Oct. 2021. doi: 10.1109/VLHCC51201.2021.9576372 2

[16] L. Zhang and S. Oney. FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality. In *Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology*, pp. 342–353. ACM, Virtual Event USA, Oct. 2020. doi: 10.1145/3379337.3415824 2

[17] J. Zhu, A. Y. Nee, and S.-K. Ong. Online Authoring for Augmented Reality Remote Maintenance. In *Signal Processing, Pattern Recognition, and Applications / 722: Computer Graphics and Imaging. AC-TAPRESS, Innsbruck, Austria, 2011*. doi: 10.2316/P.2011.722-004 2

[18] J. Zillner, E. Mendez, and D. Wagner. Augmented Reality Remote Collaboration with Dense Reconstruction. In *2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)*, pp. 38–39. IEEE, Munich, Germany, Oct. 2018. doi: 10.1109/ISMAR-Adjunct.2018.00028 1, 2