# Concept of PLM Application Integration with VR and AR Techniques

Jan Duda, Sylwester Oleszek

► To cite this version:

Jan Duda, Sylwester Oleszek. Concept of PLM Application Integration with VR and AR Techniques. IFIP International Conference on Advances in Production Management Systems (APMS), Aug 2020, Novi Sad, Serbia. pp.91-99, 10.1007/978-3-030-57997-5_11. hal-03635702

HAL Id: hal-03635702

https://inria.hal.science/hal-03635702

Submitted on 8 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Concept of PLM application integration with VR and AR tech-
niques

Jan Duda¹[0000-1111-2222-3333] and Sylwester Oleszek²[1111-2222-3333-4444]

¹ Politechnika Krakowska, al. Jana Pawła II 31-866 Kraków, Polska

² Transition Technologies PSC sp. z o.o. ul. Piotrkowska 276, 90-361 Łódź, Polska
sylwester.oleszek@gmail.com

**Abstract.** Nowadays, many industrial companies, including small and medium-sized ones, are adopting virtual manufacturing concepts to face global competition and major manufacturing challenges. The aim is to improve quality, shorten delivery time and reduce costs. However, most virtual manufacturing methods, tools and software are not integrated well enough to perform the required activities efficiently. Attention is usually focused on local and specific proficiency, thus jeopardising information exchange between departments, parallelism of work and communication along the product lifecycle. In these circumstances, the use of virtual production and digital representation of the production system and its processes becomes even more important to optimise production activities. Manufacturing industries are evolving towards digitisation, networking and globalisation. In the process of rapidly evolving information technology, digital tools and systems are being used in all industries, managing a variety of tasks throughout the product lifecycle with the use of PLM (Product Lifecycle Management) applications. PLM class systems integrate a set of applications supporting product development.

The paper presents the concept of the integration of PLM application with VR (Virtual Reality) and AR (Augmented Reality) techniques based on the example of the proprietary system integrating advanced PLM app with a mobile application in which AR technology has been implemented.

**Keywords:** Augmented Reality, Virtual Reality, Product Lifecycle Management.

## 1 Introduction

VR is a medium composed of interactive computer simulations that determine the position and actions of the participant and replace or extend the feedback to one or more his senses, giving him a sense of immersion or mental presence in the simulation, that is, in the virtual world [12]. The key elements of the ability to experience virtual reality are: virtual world, immersion, sensory feedback on user actions and interactivity [12]. Virtual reality is also a scientific and technical field that uses IT tools and behavioural interfaces to simulate the behaviour of 3D objects in the virtual
world so they can interact with each other in real time in pseudo-natural immersion
through sensomotor channels.

Augmented reality, unlike virtual reality, includes techniques for adding virtual objects to the real world [9]. Augmented reality techniques are therefore defined as those which, in real time and in a three-dimensional environment, make it possible to connect the real world with the virtual one [1,6]. In other words, the AR system is a system that connects the real world with a computer-generated world.

Virtual Manufacturing (VM) is a concept of planning production activities using virtual models and simulation tools instead of objects and their operations in the real world [4]. The way to implement this concept is a computer system that generates information about the structure, states and behaviour of the manufacturing system as observed in the natural production environment. In other words, the process of virtual production does not involve materials or physical products, but information about them. Virtual manufacturing can therefore be defined as an integrated computer model that represents the physical and logical scheme and behaviour of the actual manufacturing system [7]. Some scholars define virtual manufacturing in relation to virtual reality by describing this technology as a virtual world for manufacturing at the same time presenting virtual reality as a visualisation tool for virtual manufacturing. With this approach in mind, we can define VM as an application of integrated technologies of simulation, modelling and analysis with the use of models and virtual processes in a virtual reality environment. This integration makes it possible to identify potential problems before the actual production takes place. As a result, it is possible to improve the design of the production system, make decisions and control all stages of this process [1]. Thus, the general premise present in most definitions is that "Virtual production is nothing more than production on a computer".

The Virtual Factory (VF) paradigm is the basis for dealing with the problem of
implementing methodologies to support the design of production systems, processes,
simulation, production control, visualization and other.

The ideal implementation of a virtual factory should therefore include the creation of a comprehensive, integral, upgradeable and scalable representation of the actual factory. It should provide a virtual representation of buildings, resources, processes and products. The entire factory should be simulated as a continuous and coherent digital model that can be used throughout the entire product lifecycle - from the product idea to the final disassembly of production plants and buildings [3].

## 2 Interactive product development cycles

A modern company has to cope with the co-evolution of products, processes and production systems through strategic and operational management, promotion of engineering changes depending on market requirements and the dynamics of legislative changes. The essence of these changes is to manage the implementation of development phases based on a common model of product, process and resource data (Product, Process, Resource). Fig.1.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Management of development phases based on a common model of product, process and resource data</figcaption></figure>

One way to implement the idea of a virtual factory is to use PLM class systems. They integrate a set of applications supporting product development which include [2]:

* Product and Portfolio Management (PPM),

* Computer-aided technologies (CAx),

* Manufacturing Process Management (MPM),

* Product Data Management (PDM).

Although these systems provide support for most areas related to the planning and design phases, and effectively support product data management and process virtualization, they do not provide certain functionalities relevant to business needs, especially in the area of interoperability [10-11,12]. As a result, the whole process of virtual factory realisation cannot be carried out with the use of PLM systems in a continuous and coherent way. This is particularly visible in the lack of integration between technologies and data model used in the product design and the production systems design phases [5].

One of the important features of the product development process is concurrence. The essence of this process is the implementation of product development phases and the possibility of evaluating possible product, process and system variants in the context of the implementation of subsequent development phases. The achievement of this postulate has been made possible by separating the conceptual and detailed design stages in subsequent development stages [2]. The assessment of the obtained design solutions, apart from the application of simulation programs, may be supported by the application of VR and AR virtual manufacturing techniques. Assuming that the product development process consists of subsequent stages, where the quality of each stage depends directly on the quality of data from the previous one, the above mentioned approach allows the performance of interactive design cycles using CAx (Computer Aided x) systems and then testing using DFx (Design for x) systems, AR and VR techniques, using digital data at each individual stage. As a result, it allows to
significantly increase the speed of the whole process, and also move the detection and resolution of potential errors or problems to earlier stages. In the virtual environment process, 3D models are created and tested at a very early stage, much earlier than any physical prototypes could be produced and tested. What is more, in a virtual environment we can afford to quickly and relatively inexpensively (without the use of physical raw materials) manufacture and test many variants of both the product and the tools used for its production. We also have the ability to simulate the entire production process using various parameters [9]. This is schematically shown in Figure 2.

However, effectively designing and integrating VR and AR with Virtual Manu-facturing Systems (VMS) in such a way that they can improve production and product development processes, leading to shorter

lead times, lower costs, efficiency and quality improvement along with a clean and eco-friendly process, is a major challenge. One of the biggest difficulties to overcome is the absence of system and data interoperability, i.e. the lack of consistency in data transfer between systems (in case of many software-integrated system platforms) or between system modules performing different tasks (in case of e.g. PLM systems).

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Interactive product development cycles</figcaption></figure>

## 3 PLM system architecture integrated with the service performing tasks related to VR and AR technologies

Modern PLM systems operate according to the client-server model, and communication between system clients (the client can be a web browser, another system or system component), and the system is based on HTTP protocol, i.e. in request-response mode. Thus, it is possible to propose a generalised PLM system architecture integrated with modules performing tasks related to VR or AR technologies.

The architectural pattern of WWW application for client-server system in the most general sense assumes organizing the whole system into three layers: database, processing (business) and presentation (client) [8]. At the level of the processing layer, where PLM system services realizing business logic are located, an application performing tasks related to the preparation of data for use in VR or AR environments may be integrated. These tasks may include e.g. conversion of 3D data from native engineering formats (.prt, .CATPart, .sldprt, etc.) to a neutral or lightweight format, which enables easier transfer via HTTP protocol and easier display in mobile multimedia applications. They can also be more complex tasks, i.e. processing and prepar-
ing 3D models in such a way that they can be used directly in client applications, e.g. in mobile applications or augmented or virtual reality glasses.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Generalised architecture of the PLM system integrated with the service performing tasks related to VR and AR technologies</figcaption></figure>

Response information from the client application, e.g. in the form of metadata, can be sent directly to the PLM system business layer - for further processing - via communication protocol requests. A proposal of a generalized PLM system architecture integrated with the service performing tasks related to VR and AR technologies is presented in Figure 3.

## 4 The proposal of the procedure for technical implementation of PLM system integration with VR and AR techniques

This chapter presents the concept of the procedure for the technical implementation of the integration of the PLM commercial system with the proprietary AR system, where the integration of an advanced CAx class system with a mobile application - in which the AR technology has been applied - was accomplished. This solution enables 3D modeling activities (based on techniques adopted from product configurators) directly in AR environment. Simultaneously, in the CAx system, a parametric 3D model is created in a fully automated way [6]. The AR system consists of three components:
1. A mobile application created in Unity3D environment, in which the Vuforia augmented reality library was used,

2. CAx class system,

3. An integration module in the form of an application created with the use of the CAx system programming interface in VB.net language, the aim of which is to provide communication with the mobile application and control the automated operations of the CAx system.

The goal of the presented concept of PLM and AR systems integration is:

* implementation of centralised management of standard elements, which are used in the process of modeling and configuration, based on the principles of standards management in PLM system (permission control, versioning etc.),

* integration of the developed configuration method in the AR environment with a consistent product lifecycle management process,

* implementation of history tracking, status management and versioning of modeled product concepts during their development,

* introducing the possibility of a formalised version approval process for the product created by the customer during the configuration process.

According to the authors' intention, an integrated, uniform environment including the PLM and AR systems should allow for all configuration and modelling activities in the AR environment, as developed in [6]. However, all these activities should be performed within a PLM environment. To enable this, it is necessary to automate operations such as:

* establishing a connection between the AR and the PLM systems,

* loading elements used in the configuration process from the PLM system standard library,

* saving the 3D model developed in the process of configuration to the development container of the PLM system,

* management of the lifecycle state of the 3D model developed in the configuration process,

* end of session and logout from PLM system.

In order to implement the above mentioned actions, the following steps are necessary:

* developing a method of automatic login into the PLM system using credentials (login and password),

* creating a standard library of elements used in the configuration process in the PLM system and modifying the AR system to enable loading these elements directly from the created library,

* developing a method of automatic saving the 3D model developed in the process of configuration directly to the development container of the PLM system,

* developing a method to automatically change the state of the 3D model lifecycle developed during the configuration process (this will also enable version management).

The diagram in the BPMN notation of the product configuration process in the AR system with integrated PLM system is shown in Figure 4.
# 5 Summary

This paper presents practical definitions of virtual and augmented reality, virtual manufacturing and virtual factory. The authors highlighted that one of the possible, although not perfect, implementations of the virtual factory paradigm is using modern PLM systems. In these systems, AR and VR technologies can be implemented in the presentation layer level. However, this requires the development and integration of additional services with the PLM system, the purpose of which is to carry out tasks related to data conversion and processing in a specific way (depending on the technology and purpose), so that they can be displayed in AR or VR devices.

The paper also proposes a concept of PLM system integration with VR and AR techniques based on generalised PLM system architecture. This concept is based on the assumption that at the level of processing layer integrated with PLM system there is a software service, by means of which tasks related to VR and AR technologies are performed.

The last part of the paper presents the proposal of the procedure of technical realization of integration based on the example of a commercial PLM and a multi-module proprietary AR systems. The aim and benefits of obtaining a uniform PLM and AR environment have been defined and additional functionalities that are possible to achieve (e.g. version control, status management, management of elements used in the configuration process based on the principles of the standards library in the PLM system) have been highlighted. This part of the paper also presents actions that need to be performed in order to create a prototype version of the integration.

The authors intend to continue their work in order to implement the concept mentioned and also to examine the functionality and practical usefulness of such a solution.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Diagram of the product configuration process in the AR system integrated with the PLM system</figcaption></figure>

# References

1. Barfield, W.: Fundamentals of wearable computers and augmented reality, 2nd ed., CRC Press, NW Boca Raton, FL, United States (2015).
2. Bharath, V. G., Rajashekar, P.: Virtual Manufacturing: A Review. International Journal of Engineering Research & Technology (IJERT) NCERAME-2015 Conference Proceedings (2015).

3. Duda, J.: Zarządzanie rozwojem wyrobów w ujęciu systemowym. Cracow University of Technology (2016).

4. Fuchs, P., Moreau, G., Guitton, P.: Virtual reality: Concepts and technologies. CRC Press, NW Boca Raton, FL, United States (2011).

5. Ghielmini, G., Pedrazzoli, P., Rovere, D., Terkaj, W., Boer, C. R., Dal Maso, G., Milella, F., Sacco, M.: Virtual Factory Manager for semantic data handling. CIRP Journal of Manufacturing Science and Technology, vol. 6, pp. 281-291 (2013). doi:10.1016/j.cirpj.2013.08.001

6. Januszka, M.: Metoda wspomagania procesu projektowania i konstruowania z zastosowaniem poszerzonej rzeczywistości. PhD thesis, Silesian University of Technology, Gliwice (2012).

7. Keyvani, A., Danielsson, F.: A Virtual Manufacturing Approach for Integrating Fixture Design with Process Planning. In: Huang G.Q., Mak K.L., Maropoulos P.G. (eds) Proceedings of the 6th CIRP-Sponsored International Conference on Digital Enterprise Technology. Advances in Intelligent and Soft Computing, vol 66, pp. 483-496. Springer, Berlin, Heidelberg (2010).

8. Lafleur, M., Terkaj, W., Belkadi, F., Urgo, M., Bernard, A., Colledani, M.: An Onto-Based Interoperability Framework for the Connection of PLM and Production Capability Tools. pp. 134-145 (2016). doi:10.1007/978-3-319-54660-5-13

9. Mann, S.: Intelligent image processing. Adaptive and learning systems for signal processing, communications, and control. IEEE, New York, NY, United States (2002).

10. Oleszek, S., Metoda wspomagania projektowania naczyń szklanych z zastosowaniem konfiguratora w środowisku poszerzonej rzeczywistości. PhD thesis, Silesian University of Technology, Gliwice (2018).

11. Sharma, P.: Concept of Virtual Manufacturing. GRD Journals - Global Research and Development Journal for Engineering, vol. 2, issue 6, pp. 285-292, May (2017).

12. Sherman, W.R., Craig A.: Understanding virtual reality: Interface, application, and design. The Morgan Kaufmann Series in Computer Graphics, San Francisco, CA, United States (2002).

13. Sienkiewicz, J. E., Syty, P.: Architektura warstwowa aplikacji internetowych. Oblicza Internetu. Conference Proceedings, PWSZ, Elbląg (2008).

14. Souza, M., Sacco, M., Porto, V.: Virtual manufacturing as a way for the factory of the future. Journal of Intelligent Manufacturing, vol. 17, pp. 725-735 (2006). doi:10.1007/s10845-006-0041-1

15. Terkaj, W., Pedrielli, G., Sacco, M.: Virtual Factory Data Model. CEUR Workshop Proceedings, vol. 886, pp. 29-43 (2012)

16. Tolio, T., Ceglarek, D., ElMaraghy, H.A., Fischer, A., Hu, S., Laperrière, L., Newman, S., Váncza, J.: SPECIES - Co-evolution of Products, Processes and Production Systems. CIRP Annals - Manufacturing Technology, vol. 59, No. 2, pp. 672-693 (2010). doi:10.1016/j.cirp.2010.05.008

17. Tolio, T., Sacco, M., Terkaj, W., Urgo, M.: Virtual Factory: an Integrated Framework for Manufacturing Systems Design and Analysis. Forty Sixth CIRP Conference on Manufacturing Systems (2013).