<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Design and Development of a G-Code Generator for CNC Machine Tools Based on Augmented Reality (AR)

Dimitris Mourtzis✉, Panagiotis Kaimasidis, John Angelopoulos✉, and Nikos Panopoulos✉

Laboratory for Manufacturing Systems and Automation, Department of Mechanical Engineering and Aeronautics, University of Patras, 26504 Rio Patras, Greece

mourtzis@lms.mech.upatras.gr

**Abstract.** Modern manufacturing systems rely on the utilization of Computer Numerical Control (CNC) for the manufacturing of products and components. By extension, engineers must be capable of programming machine tools (through G-code) in order to achieve the desired results. However, this process is time consuming and requires highly skilled and trained personnel. On the other hand, under the Industry 4.0 framework, several cutting-edge digital technologies have been developed in an attempt to improve user perception, such as Extended Reality (XR), which encapsulates Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). Furthermore, the above-mentioned technologies enable engineers to extend Human-Machine Interaction (HMI) into new dimensions. Therefore, in this research work the design and development of a framework for facilitating engineers during the generation G-code script for machining operations based on the utilization of AR is presented. The MR functionalities will facilitate develop a better perception of the machining processes, as well as support the generation of virtual instructions for shop-floor operators. Further to that, advanced communication functionalities are provisioned based on Cloud technologies, in order to automate the transmission of G-code scripts/instructions to the machine tools.

**Keywords:** Mixed Reality (MR) · Computer Numerical Control (CNC) · G-code · Human-Machine-Interaction (HMI)

## 1 Introduction

Towards the Mass Personalization era manufacturing companies aim to reduce the length and cost of the product development cycle [1]. A paradigm shift in manufacturing from ‘real’ to ‘virtual’ production has resulted in the field since the early 1990s. However, with the pillar technologies of Industry 4.0, it is now possible to simulate some of the activities of a physical manufacturing system. The main goal is to understand and simulate the behavior of a specific manufacturing system prior to physical production, reducing the amount of testing and experiments on the shop floor [2]. Trial and error is frequently
used in the planning of manufacturing processes. Engineers use physical experimen-
tation to verify alternative plans in the case of complex processes. This approach fre-
quently prevents engineers from achieving the desired cycle time and cost reductions
[3]. Additionally, less material is wasted and interruptions to the actual machine on the
shopfloor can be avoided by using a virtual system (Owen JV, 1997). Moreover, there
is less concern about safety issues and cloud-based collaboration can be achieved for
education and training purposes [4]. Focusing on Computer Numerical Control Systems
(CNC) and more recently to Cyber Physical Machine Tools [5], part programming can
be accomplished either manually or using automated methods such as Computer Aided
Manufacturing (CAM) [6]. In Smart Manufacturing landscape, there is a critical need to
have appropriate tools of human-machine interaction (HMI) to support customization
and personalization needs [7]. Therefore, the following research question arises: "what
should the HMI system look like that individual can access and interact within Machine
Tools to design and manufacture customized products? "Thus, the contribution of this
research work is focused on the design and development of a method for the visualization
of G- and M- code scripts, based on the utilization of Cloud technologies and XR. The
MR functionalities will facilitate develop a better perception of the machining processes,
as well as support the generation of virtual instructions for shop-floor operators. Finally,
the proposed framework has been validated in an industrial CNC milling machine.

The rest of the paper is structured as follows. In Sect. 2 the most relevant and recent related research works are investigated. Then, in Sect. 3 the architecture of the proposed method is presented and discussed. Then, In Sect. 4 the implementation of the method is presented. Finally, in Sect. 5 conclusions are drawn, and points for further research are discussed by the authors.

## 2 State of the Art

Voxels can be realized as the equivalent of pixels in the 3D space, forming perfect
cubes [17]. During the last decades various applications in the field of Extended Reality
(XR) as well as in 3D rendering, voxel-based architectures have been implemented.
A classic example is the HP (Hewlett Packard) 3D printer, which uses voxels sized at
25-micron building blocks (for reader's reference the size of the voxels is equivalent of
1/4 the thickness of a single human hair). Voxel-based architectures can be found in the
literature in two main categories, either as regular voxel grids, or as octree structures. A
variety of visualization methods and techniques, in particular i) raytracing, ii) topological
analysis, iii) volume estimation, iv) collision detection, are based on voxels in an attempt
to efficiently process more complex 3D geometries [8]. The importance of voxel-based
architectures is underlined by the fact that they are simple representations of the 3D
volumes, instead they can carry additional information about the volumes, mimicking
the structure of real-world objects (e.g. inclusion of material properties). Therefore,
voxels can be realized as data structures capable of carrying semantic labels, facilitating
engineers to model 3D systems in a more natural/efficient way [9]. In the available
literature there are several examples of voxel engines. Among them VoxelPrint, which is
a Grasshopper plug-in is focused on the simulation of 3D concrete printing constructions
as presented in [10]. Nie et al. in [11] presented a voxel-based method is proposed for
enabling the calculation of accuracy regarding Material Removal Rate (MRR), based on graphics processing units (GPGPU) computing. Furthermore, similar approaches have been proposed by the authors [10, 11] to improve the simulation of five-axis milling processes by calculating and visualizing the material deformation induced by the cutting force exerted on the workpiece. As such, voxel-based architectures can be utilized to improve the simulation, visualization and modelling of systems, in order to represent more closely their natural behavior. By extension, simulation techniques [12] and voxel techniques need to be further examined as a coupled system.

On the other hand, Computer Aided Manufacturing (CAM) which is based on the visualization of the G- and M- code, can be further improved by new digital technologies, such as AR, in order to facilitate engineers visualize in a more intuitive manner the various material removal processes and whenever possible to optimize the process parameters. To that end, virtual instructions were overlayed in the user's AR Head Mounted Display (HMD) field of view, while virtual CAD models were displayed around physical objects [13]. Furthermore, AR has been investigated for interacting with robots for path planning [14], spatial programming [15] and trajectory planning [16]. Similarly, AR has been used to simulate manufacturing processes. Various manufacturing processes, such as metal casting [17] and CNC machining [18], have been simulated in an AR environment for process plan verification, material cost reduction, and novice mechanist training [19]. However, the significant contribution of this research work compared to the abovementioned is that the MR functionalities will develop a better perception of the ma-chinning processes, as well as support the generation of virtual instructions for shop-floor operators. Further to that, advanced communication functionalities are pro-visioned based on Cloud technologies, in order to automate the transmission of G-code scripts/instructions to the machine tools.

Lin and Fu in [20] developed the Virtual Machine Tool Structural Modeller (VMSM) system, which can be used to model machine tools in a virtual reality environment. The architecture of the system is made up of several modules, including component modules, module shape libraries, combination rule libraries, and structure libraries. The system allows a user to choose the structure of the machine tool, modify it, control it, and simulate the machining process. Moreover, Training is a crucial part of the current challenging production environment. Kao et al. [21] developed the VRCNC system to support virtual training. The VR CNC structure and the VR CNC controller were built as two separate modules. The machine table, servomotor, driving system, and feedback system are all part of the VR CNC structure. Next, in the development of a VR-based training system, both immersion and desktop VR can be used. One such system in literature is the Products Virtual Analysis System (PVAS) [22]. There were four main functions used. They are as follows: (a) virtual reality with stereo glasses, (b) product kinematics analysis and optimization, (c) dynamics, and (d) a virtual interactive environment for virtual machining and assembly processes.

## 3 Architecture of the Proposed Method

In this Section of the manuscript the general system architecture will be presented and discussed. Consequently, it is necessary to break-down the architecture into key modules,
and by extension to analyze the information flow. In Fig. 1 the general system architecture is illustrated.

<figure><img src="image_2.png" /><figcaption>Fig. 1. General system architecture</figcaption></figure>

## 3.1 Cloud Platform and Database

The Cloud platform can be realized as a virtual workspace for hosting the services required for the operation of the proposed method. More specifically, in the Cloud platform, there has been provisioned a space for the handling of users, registration and login operations. Secondly, on the Cloud platform there have been developed several libraries, containing cutting tools (end-mills) along with the 3D CAD of each cutting tool, so that they can be retrieved during the generation of the G- and M- codes. This service is entitled “Tool Handling Service - THS”. The THS also supports the addition of new tools through the corresponding user interface. At this point, it is stressed out that the 3D geometries of the cutting tools have been designed fully parametrically, in order to further minimize the space requirements on the Cloud platform. In the following Fig. 2 the modelling of the end mills implemented in the architecture is presented.

<figure><img src="image_3.png" /><figcaption>Fig. 2. UML diagram for end mills</figcaption></figure>

## 3.2 Voxel Generator

As a recap of the previous paragraphs, voxels can be realized as 3D pixels. The application was created using Unity by Unity Technologies. Unity is a game development tool that
also provides us with the ability to create 3D applications in both AR &VR. In our case, Unity AR Foundation is utilized. Other systems could have been used, but Unity provides with a much friendlier and more efficient user interface, making development easier and highly modifiable. The “AR Tracked Image” in Unity corresponds to the real-world position of the preloaded image to that of the “parent” cube. A 3D mesh of voxels is generated using C#. Voxels provide a digital simulation of the real-world analog behavior of the workpiece. The realism of the simulation is in direct correlation with the computational capabilities of each device, as a more realistic simulation requires more and smaller voxels, which comes at the cost of computational resources. The size of the voxels in turn defines the scale of the cutting tool along with the real-world scale of the simulation. In the current implementation of the proposed method the applied voxel size has a volume of 1.0*E-9 m³. Each voxel has a cube collider (triggered) attached to it allowing Unity Engine to recognize collisions between the cutting tool (which has “Rigidbody” property) and the 3D geometry of the material. Consequently, whenever collision is detected, the colliding voxels are deactivated. Even though using colliders is not the most efficient way of recognizing collisions between the cutting tool and each voxel, they are necessary as every tool has a deferent geometry which cannot be calculated by a generic algorithm for every type of tool. Additionally, the utilization of voxels provides the user with access to easily interchangeable materials to simulate different workpieces, in the following examples Aluminum is used. The material can be altered by changing the Unity material properties of the “parent” cube. Regarding the memory allocation, each the voxel requires a minimum of 1 byte, for a workpiece of 1.0 m*0.5 m*0.5 m (25*E-2 m³), constantly 0.232831 Gb of RAM memory is the minimum requirement for the voxel storage. Consequently, in an attempt to avoid the possibility of memory drain (taking into consideration the use of mobile devices), voxels are grouped in chunks, with a chunk size of 16×16×16 voxels which requires 4096 bytes (4 MB) of RAM memory. Furthermore, the access to a specific voxel can be facilitated by the storage of the voxel chunks in a serial array, requiring only knowledge of the chunk ID and the voxel ID. Finally latency is an issue that should be addressed, and we have so far adjusted the framerate to minimize the problem, until further development.

## 3.3 Augmented Reality Module

The Augmented Reality (XR) module is one of the key modules of the proposed method since it constitutes the Human-Machine Interface (HMI). Through the AR module the users are capable of interacting with the virtual components and setup the CAM instructions, based on the utilization of intuitive Graphical User Interface (GUI) elements. When the user begins a new session, the first selection is the type of material to be used. Then the initial dimensions as well as the form of the billet/casting are inserted. Upon selection of the basic parameters, the Image recognition module is activated allowing the XR visualization to appear on the user's Field of View (FOV). The material as well as the Cutting tool is then displayed on the table of the machine tool (or chuck, wherever applicable). When the user taps on “GCode Input” a panel is activated that allows the serial input of G-commands. Upon pressing continue the panel is deactivated and the simulation is realized. The material removal processes is simulated by disabling the voxels when they collide with the cutting tool. Unity 3D physics engine is responsible
for detecting the collisions using “Rigidbody” properties. In the flowchart presented in Fig. 3, the main modules of the architecture are presented, highlighting the interactions and the information flow.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Flowchart of the proposed method</figcaption></figure>

Regarding the content of Fig. 3, (A) represents the scene generation, and (B) represents the G-/M-Code simulation in AR. Concretely in (1) the users inputs are gathered as presented in Fig. 5 (1, 2). Then in (2) the cutting tool prefab and name are predetermined
by the user in selection scene, as shown in Fig. 5 (2). The G-code commands are input by the user in simulation scene, as shown in Fig. 5 (3). Finally the G-/M-Code is input as a list and the tool simulation is visualized (refer to Fig. 5 (3)).

# 4 Implementation

The first part of the implementation process requires the connection of the AR tool with the Cloud platform in order to gain access to the initial CAD files of the part to be produced, and the cutting tools/inserts available for the CNC milling machine. The initial billet is loaded as a geometry in a suitable file format (Collada file). Afterwards, a list of available G- and M-Operations is loaded from the Cloud database. Further to that, in an attempt to assist the users to setup the machine correctly, a set of AR steps for the correct installation of the cutting tools. It is stressed out that the tool, automatically detects the tool based on its name, thus the corresponding steps and specifications are loaded and displayed on the user's FOV.

<figure><img src="image_5.png" /><figcaption>Fig. 4. 1) Material selection GUI; 2) End mill selection GUI; 3) G-/M- code command insert GUI</figcaption></figure>

For the development of the AR module the Vuforia Engine API (Application Programming Interface), which provides the necessary algorithms for the environmental understanding and the registration of the 3D information in the user's FOV. In order to provide a better/more intuitive user experience tool animations have also been implemented, following the path as it is generated by each G- command. The development of
the necessary GUIs (Graphical User Interfaces) has been accomplished via the utilizat-
tion of the Unity 3D game engine (See Fig. 4). The code scripts have been developed in
C# with the use of Microsoft Visual Studio IDE (Integrated Development Environment).
The current implementation of the proposed method is compatible with smart mobile
devices based either on Android or iOS operating systems.

5 Experimental Case Study

The applicability of the developed method is tested on a use case of a CNC milling
machine. Currently, the engineers use standard CAM tools in order to generate the code
required for the CNC milling machine. Although the current approach is functional,
expert, well-educated/experienced engineers are required in order to setup processes
for the CNC milling machine. However, the manufacturing company has identified that
less-experienced engineers should be trained in order to be capable of producing such
code script, as well as to facilitate more experienced users to visualize the process and
eliminate any errors (such as use of proper tooling, proper clamping/securing the part
on the machining table etc.) which could be facilitated via the proposed approach.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Simulation of milling operation via AR</figcaption></figure>

6 Concluding Remarks and Future Work

The Virtual Machine Shop Application for G- and M- codes import and visualization in AR environment was created to aid in the assessment of various decision variables in a virtual manufacturing environment. For example, the user could stabilize the virtual
workpiece using real world work holding devices, as shown in Fig. 5. It enables a design, verification, and redesign loop to be carried out in a manner similar to that used in the real world. In addition, the intuitive interaction between the human and the process allows for the reliable verification of human-related parameters. The proposed framework can also be used to aid in the early stages of machine tool operator training. This simulation environment provides realistic and safe training conditions for immersive and interactive hands-on practice with the process, in contrast to physical training, which can be dangerous and expensive. Furthermore, AR allows for far greater adaptability, as the trainee practices on the real-world machine, making the educational experience for effective. The user, before the utilization of the application must follow the same safety routine that he would follow if he was using the real machine. That empowers the educational purpose of this AR application and builds important safety habits to the user. Finally, the smartphone version is a better option than a desktop one as they are cheaper, more widespread and portable [23].

To summarize, in this research work the development of a method for the composition of G- and M- codes based on AR visualizations. The key outcome of the presented research work is the development of a mobile application for the support of generation of machine code to aid in manufacturing processes and improve education of new engineers in CAM. Through the AR visualizations, engineers are capable of visualizing each step of the process, including the motion of the cutting tool as well as the removal of material. Along these lines, engineers be actively involved in full automation of the production segment of PLM.

Future research work will be focused on the implementation of physics in order to enable the accurate visualization of chip formation. An additional step will be the support of Head Mounted Displays (HMD).

## References

1. Mourtzis, D.: The mass personalization of global networks. In: Design and Operation of Production Networks for Mass Personalization in the Era of Cloud Technology, pp. 79–116 (2022). https://doi.org/10.1016/B978-0-12-823657-4,00006-3

2. Kadir, A.A., Xu, X., Hämmerle, E.: Virtual machine tools and virtual machining—a technological review. Robot. Comput. Integr. Manuf. **27**(3), 494–508 (2011). https://doi.org/10.1016/j.rcim.2010.10.003

3. Chryssolouris, G., Mavrikios, D., Fragos, D., Karabatsou, V., Pistiolis, K.: A novel virtual experimentation approach to planning and training for manufacturing processes—the virtual machine shop. Int. J. Comput. Integr. Manuf. **15**(3), 214–221 (2002). https://doi.org/10.1080/09511920110034978

4. Owen, J.V.: Virtual manufacturing. Manuf. Eng. **119**, 84–90 (1997)

5. Liu, C., Xu, X.W.: Cyber-physical machine tool – the era of machine tool 4.0. Procedia CIRP **63**, 70–75 (2017). https://doi.org/10.1016/j.procir.2017.03.078

6. Mourtzis D., Makris S., Chryssolouris G.: Computer-aided manufacturing. In: Chatti S., Laperrière L., Reinhart G., Tolio T. (eds) CIRP Encyclopedia of Production Engineering. Springer, Berlin, Heidelberg (2018). https://doi.org/10.1007/978-3-642-35950-7_6550-4

7. Why Multi-Material Additive Manufacturing Will Change Our Approach To Part Design. https://aerosint.com/why-multi-material-additive-manufacturing-will-change-our-approach-to-part-design/. Accessed 14 Mar 2022
8. Gebhardt, S., Payzer, E., Salemann, L., Fettinger, A., Rotenberg, E., Seher, C.: Polygons, point-clouds and voxels: a comparison of high-fidelity terrain representations. In Simulation Interoperability Workshop and Special Workshop on Reuse of Environmental Data for Simulation—Processes, Standards, and Lessons Learned, p. 9 (2009)

9. Regenbrecht, H., Park, J.W., Ott, C., Mills, S., Cook, M., Langlotz, T.: Preaching voxels: an alternative approach to mixed reality. Frontiers ICT 6(7), 7 (2019). https://doi.org/10.3389/fict.2019.00007

10. Cherabier, I., Häne, C., Oswald, MR., Pollefeys, M.: Multi-label semantic 3D reconstruction using voxel blocks. In: 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA, pp. 601–610 (2016). https://doi.org/10.1109/3DV.2016.68

11. Nie, Z., Lynn, R., Tucker, T., Kurfess, T.: Voxel-based analysis and modeling of MRR computational accuracy in milling process. CIRP J. Manuf. Sci. Technol. 27, 78–92 (2019). https://doi.org/10.1016/j.cirpj.2019.07.003

12. Mourtzis, D.: Simulation in the design and operation of manufacturing systems: state of the art and new trends. Int. J. Prod. Res. 58(7), 1927–1949 (2020). https://doi.org/10.1080/00207543.2019.1636321

13. Makris, S., Pintzos, G., Rentzos, L., Chryssolouris, G.: Assembly support using ar technology based on automatic sequence generation. CIRP Ann. Manuf. Technol. 62(1), 9–12 (2013). https://doi.org/10.1016/j.cirp.2013.03.095

14. Fang, H.C., Ong, S.K., Nee, A.Y.C.: A novel augmented reality-based interface for robot path planning. Int. J. Interact. Des. Manuf. (IJIDeM) 8(1), 33–42 (2013). https://doi.org/10.1007/s12008-013-0191-2

15. Lambrecht, L., Kleinsorge, M., Rosenstrauch, M., Kruger, J.: Spatial programming for industrial robots through task demonstration. Int. J. Adv. Robot. Syst. 10(5), 254 (2013). https://doi.org/10.5772/55640

16. Fang, H., Ong, S., Nee, A.: Interactive robot trajectory planning and simulation using augmented reality. Robot. Comput. Integr. Manuf. 28(2), 227–237 (2012). https://doi.org/10.1016/j.rcim.2011.09.003

17. Watanuki, K., Hou, L.: Augmented reality-based training system for metal casting. J. Mech. Sci. Technol. 24(1), 237–240 (2010). https://doi.org/10.1007/s12206-009-1175-9

18. Zhang, J., Ong, S.-K., Nee, A.Y.: Development of an AR system achieving in situ machining simulation on a 3-axis CNC machine. Comput. Anim. Virtual Worlds 21(2), 103–115 (2010). https://doi.org/10.1002/cav.327

19. Mourtzis, D., Zogopoulos, V., Katagis, I., Lagios, P.: Augmented reality based visualization of CAM instructions towards industry 4.0 paradigm: a CNC bending machine case study. In: 28th CIRP Design Conference 2018, Procedia CIRP, pp. 368–373 (2018). https://doi.org/10.1016/j.procir.2018.02.045

20. Marinov, V.P., Seetharamu, S.: Virtual machining operation: a concept and an example. In: Intelligent Systems in Design and Manufacturing, vol. 5605, pp. 206–213 (2004). https://doi.org/10.1117/12.571333

21. Kao, Y.C., Chen, H.Y., Chen, Y.C.: Development of a virtual controller integrating virtual and physical CNC. In: Materials Science Forum, vol. 505, pp. 631–636 (2006). https://doi.org/10.4028/www.scientific.net/MSF.505-507.631

22. Wang, J., Chen, E., Yang, J., Wang, W.: Applications of virtual reality in turn-milling centre. In: 2008 IEEE International Conference on Automation and Logistics, pp. 2302–2305 (2008). https://doi.org/10.1109/ICAL.2008.4636550

23. Doolani, S., et al.: A review of extended reality (XR) technologies for manufacturing training. Technologies. 8(4), 77 (2020). https://doi.org/10.3390/technologies8040077