# Smart Products in Smart Manufacturing Systems: An Opportunity to Utilize AR?

Joshua Gross, Thorsten Wuest

► To cite this version:

Joshua Gross, Thorsten Wuest. Smart Products in Smart Manufacturing Systems: An Opportunity to Utilize AR?. IFIP International Conference on Advances in Production Management Systems (APMS), Aug 2020, Novi Sad, Serbia. pp.487-494, 10.1007/978-3-030-57993-7_55. hal-03630878

HAL Id: hal-03630878

https://inria.hal.science/hal-03630878

Submitted on 5 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Smart Products in Smart Manufacturing Systems:
## An Opportunity to Utilize AR?

Joshua Gross¹ and Thorsten Wuest¹

¹ West Virginia University, Morgantown WV 26505, USA
jjg0016@mix.wvu.edu, thwuest@mail.wvu.edu

**Abstract.** The evolution of manufacturing throughout time has allowed our in-dustrial world to develop and ever faster create various products in large numbers using a myriad of industrial manufacturing processes. However, like everything in the world, time moves forward and the evolution does not stop but accelerate. Therefore, the implementation of Augmented Reality (AR) through the use of the internet of things, low cost sensors, and free programming languages, are all contributing factors to further advanced the industry. These implementations will allow for faster data processing, quicker error recovery time, loss and error prevention, a wider range of employees at different skill levels to be productive in a manufacturing environment. The implementation of an Arduino based sensor system into the manufacturing process enables data streaming to an AR device. This system allows for instant data streaming and visualization to the user/operator with information pertaining to the status of the product ('state'), key developments, and changes in the product/or system, any associated errors, and the best way to proceed with resolving them. The AR enabled system with its capability to streaming relevant information prevents losses associated with product quality issues and/or process down-time among a myriad of other applications. Preliminary research shows the ability to collect data from the product in addition to data processing on an external device to later be used on an augmented reality device.

**Keywords:** AR, Augmented Reality, Smart Glasses, Smart Manufacturing, Smart Product, Industry 4.0.

## 1 Introduction

The ongoing fourth industrial revolution is enabled by data, data processing, and connectivity mainly through the internet of things [1]. Digitalization of data and processes aims to decrease production time, prevent errors and mistakes, and facilitates rapid designing and prototyping of one-of-a-kind products [2]. The emergence of smart manufacturing will allow for all of these to occur while enabling individuals and companies to oversee their entire manufacturing process. This monitoring of the process enables this to be done in high detail, remotely, and with minimal effort. Real-time in-situ data collection and analysis is key to the future of manufacturing and this project aims to contribute to the progress in this field through an example of a novel in-situ data collection application using smart products during the production process itself. As
technology further develops, systems such as the one explained in this paper can collect data in real-time and then be used in a visualization method, such as AR. [3]. The majority of smart manufacturing systems collect data of the to-be-manufactured part via sensor systems integrated in the machine tool and/or assembly line [11]. In this case, the to-be-manufactured smart product collects the manufacturing process data via a system integrated sensor system. The idea is that the smart product collects data of the manufacturing process in real-time, and the insights from the analyzed data is provided to the operator as decision support via an Augmented Reality (AR) application.

The integrated sensor system allows for rapid and specific data analysis pertaining to individual to-be-manufactured products by shop floor employees. In large scale manufacturing processes, product errors and production line down time can cost companies large amounts of money [4]. Having manufacturing floor employees overseeing the manufacturing process and products with AR is expected to contribute to the prevention and resolution of product associated errors [5, 13]. The aim of this research is to demonstrate the application of AR in combination with a smart-connected product within the manufacturing process and show that the AR integration allows for rapid product data observance in addition to error resolution via a test-bed cyber-physical production system.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Testbed setup – CPlab Smart Manufacturing System</figcaption></figure>

Our experimental setup and development of the Arduino based sensor system is detailed in the following, touching upon how it is utilized in order to achieve data collection, analysis, and user feedback. The research is conducted in a test-bed comprised of a FESTO Didactics cyber-physical laboratory (CP-lab) (see Fig. 1). The eight-stage smart manufacturing system manufactures a simulated cell phone case that consists of two separate plastic parts that are manipulated and assembled in a fully automated manufacturing process. This data and analysis allows us to instantly track the progress and working state of a single product in order to give the operator instant feedback.
However, this test-bed application can be theoretically applied to a much larger scale
in many different scenarios such as automotive part monitoring, machine and part
maintenance, and quality control of products. The data processing combined with the
AR applications allows the operator to instantly and easily read out all of the pertinent
information regarding the part that they are inquiring into.

The paper is structured as follows: in section 2, a brief overview of the background and state of the art in smart product based process monitoring and AR applications in manufacturing is provided. Following, the research methodology is depicted in section 3 before initial results are presented in section 4. The future applications of the setup are discussed in section 5 before section 6 concludes the paper and provides a short outlook on future research.

## 2 Background and State of the Art

AR offers significant potential with regard to improving the overall manufacturing system and its processes [12]. Currently, AR is being utilized in the manufacturing process to improve quality control, maintenance, and assembly. Within the last three years, AR has been directly applied to the following industries: medical, military, robotics, and manufacturing [2]. Porsche, for example, is actively using AR to ensure that manufactured parts meet the design expectations and dimensions. By simply looking at a part, the AR glasses will “instantly gauge the dimensional accuracy, surface finish, tolerances and interference and other potential issues” [6]. Continually, Boeing uses AR to assist manufacturing floor employees with the electrical components installation to ensure accuracy and decrease installation time. Furthermore, the use of AR in this application also ensures the employee are not missing any components [7].

Another identified current AR application in manufacturing is utilized by ThyssenKrupp, a large industrial company among others in the elevator industry. AR is used to scan a staircase to measure all dimensions which are instantly sent to the manufacturing department. Furthermore, the AR allows the customer to see what the stair lift will look like after installation [6]. In the healthcare industry, GE Healthcare is using AR for “projecting the work instructions onto the parts and use sensors to monitor the assembly and give feedback to the operator” which inherently increases accuracy and efficiency [8]. With the implementation of AR, “activities, such as design, planning, machining, etc., are done right the first time without the need for subsequent re-works and modifications” and therefore enables error prevention [2]. Cyber-Physical systems such as the laboratory set-up used in this study enable a more accurate manufacturing process through the internet of things as well as AR. While a one-of-a-kind part is being produced, “the part is checked and verified against product design data in particular its dimensions and tolerances” which ensures that the part is being created to the proper specifications [9]. Furthermore, Anderl states, “as products or components of products are based on cyber-physical systems their smart sensors are able to deliver data about the products’ or the components’ condition”. If this data is sent to an AR system for analysis, the worker can utilize the information in real-time and “predictive maintenance can be taken” [9]. AR would allow any worker to observe the part during
the manufacturing process while being shown how the part being produced compares to the design and specifications. Continually, the construction industry is currently utilizing AR “to address defects that might be overlooked in the inspection process” [10]. This same technology can be applied to the manufacturing industry as a way to constantly monitor the part being produced. This constant observance in addition to the data display through AR will therefore prevent errors as well as help correct any existing ones. These examples of current AR applications demonstrate its potential when being fully implemented into the manufacturing process at all stages to improve efficiency, accuracy, and safety.

The current laboratory set up used for this study includes an Arduino Nano based sensor system which allows for high customization to collect data on various attributes of the product during production. As of now, the sensor system is capable of tracking acceleration and magnetic field, has a toggle switch for tracking mode, and a SD card reading to store all of the collected data. Figure 2 shows a CAD rendition of the mock cell phone with the Arduino Nano and associated sensors used to collect said data.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Arduino Nano Sensor System CAD Layout</figcaption></figure>

Due to limitations within the Arduino Nano and space constraints within the product's volume, the data is currently not processed in real-time. Off-the-shelf sensors are easier to use and deploy but have high physical and programmatic complications when it comes to customization of the solution. A custom-made Arduino based sensor system allows for high customizability at a low cost.

The implementation of the sensor system into the CP-lab system allows for customized data tracking throughout the entire manufacturing process whereas most current industries have the ability to strictly track physical location or status. For example, our sensor system in the CP-lab system will allow the user to see how the part is oriented, which station it is at, the temperature, and current progress through the manufacturing process whereas current tracking methods would simply read out how far along the part is in the manufacturing process. Table 1 shows a comparison of different sensor systems used for tracking in industry. This table compares barcodes/data matrices, radio
frequency identification (RFID), off-the-shelf sensors (OTS), and our Arduino sensor system. Barcodes or data matrices are patterns that encode data that can be read by a scanner therefore showing the user the encoded data. RFID tags send data regarding a product through the use of radio waves and an antenna. If these tags have their own power source, they are categorized as being 'active', otherwise they are 'passive'. The OTS sensor is available to anyone and can be purchased online and comes with capabilities determined by the manufacturer. Based on this table it is clear that the Arduino system has the most potential and capability. The boxes that are marked with a '(x)' indicates a sort of caveat. Regarding the passive RFID, this has the ability to locate if and only if it is connected to the machine itself. In its own, the passive RFID does not have the ability to locate items. The OTS sensor is customizable to a certain degree. The OTS can be customized within the spectrum that the manufacturer allows. Outside of the guidelines set by the manufacturer, the OTS cannot be customized.

<table><caption>**Table 1.** Comparing popular tracking technologies with regard to data processing capabilities</caption><thead><tr><th></th><th>Barcode /<br>Data Matrix</th><th>RFID<br>(Passive)</th><th>RFID<br>(Active)</th><th>OTS</th><th>Arduino Sensor Sys-<br>tem</th></tr></thead><tbody><tr><td>Identify</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td></tr><tr><td>Identify + Locate</td><td>x</td><td>(x)</td><td>x</td><td>x</td><td>x</td></tr><tr><td>Identify + Locate<br>+ Sense</td><td>-</td><td>-</td><td>x</td><td>x</td><td>x</td></tr><tr><td>Identify + Locate<br>+ Sense + Com-<br>municate</td><td>-</td><td>-</td><td>-</td><td>x</td><td>x</td></tr><tr><td>Identify + Locate<br>+ Sense + Com-<br>municate + Pro-<br>cess Data</td><td>-</td><td>-</td><td>-</td><td>-</td><td>x</td></tr><tr><td>Customizable</td><td>-</td><td>-</td><td>-</td><td>(x)</td><td>x</td></tr></tbody></table>

### 3. Case Study of Smart Manufacturing Testbed

The Arduino Nano based sensor system is directly placed inside the beginning half of the product. As the product goes through the different stages of placement, drilling, laser measuring, turning, heating, compression, and sorting, data is collected and placed onto the SD card. The data is in the form of an Excel file which is then imported into Python. In Python, the data can be processed in practically any fashion before continuing. Once the raw data has been processed and analyzed, it is then embedded into an Android compatible graphic user interface (GUI) such as Kivy, where is it made tangible and can be interacted with. This GUI is what presents the data in a user-friendly fashion to the user and data can be sorted and organized. Due to the Android compatibility, the GUI can be compressed into an installer executable and placed onto the Android based AR glasses where the user will directly and most practically be interacting
with the processed data. Figure 3 illustrates the principle structure of the project and how all of the steps lead to enabling better decision making for the user, the manufacturing engineer, on whether or not to take action.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Theoretical Project Work Flow</figcaption></figure>

Due to the nature of Arduino single board computer systems, the sensor system is highly customizable and can be changed easily to collect different data types. Furthermore, Arduinos allow for the pre-processing of data such as ignoring certain factors, actively looking for outliers or false data, or searching for anticipated data. Having ‘clean’ data coming out of the sensor system enables faster, easier, and more accurate results and feedback to come out of the Python data processing code.

## 3 Results

The results so far reflect the current stage of development of the ongoing research project. At this point, the smart-connected product is capable of actively and reliably collecting data during the manufacturing process. The acquired sensor data contains multiple sensor measurements and visualization of the data enables an analysis of the smart-connected products progress and changing state through the multi-stage smart manufacturing system. However, the wireless connectivity to the AR glasses and/or network is still development. While the Arduino system provides several advantages (see Table 1), it is bulkier and difficult to fully integrate in the product itself. Therefore, certain adjustments to the manufacturing process itself were necessary.

Some findings at this stage include: struggles in data transfer between devices, inability to access certain features on the AR glasses, and the inability to extract useful data off of the CP-lab system. However, the sensors system has been able to collect data containing elapsed time, 3-axis acceleration, orientation, and the magnetic field. This data is compiled into an MS Excel sheet on an SD card to be used for data processing. Note that this data flow is not wireless due to limitations with the Arduino Nano which is not Bluetooth capable at the moment. Furthermore, using the current sensor system makes it difficult to collect data while in the CP-lab system as the wiring
interferes with the laser sensors thus sending a false signal to the system and therefore
making a faulty part.

Due to challenges faced in multiple stages of our ongoing research project, there is
limited tangible data available as of this point in time. Other limitations include unex-
pected barriers, such as ability to gain developer access on the AR glasses thus limiting
testing ability with the connection between the data and the glasses. Continually, this
prevents the installation of any custom made software to be placed onto the glasses and
therefore the data processing code cannot be directly utilized on the glasses. Another
limitation includes the lack of available space inside of the manufactured part which
limits the number and size of sensors to be used.

4 Conclusions and Future Work

The initial results of this project show promise with regard to providing a richer data picture seamlessly to the operator via AR. However, there are several challenges that need to be addressed to further the research. Certain obstacles can be overcome relatively quickly whereas others will take a lot more time and resources. For example, redesigning and developing the sensors system will prevent the interference with the laser measuring system and therefore prevent false signals to be sent to the CP-lab system, a relatively easy solution. However, other issues such as the lack of access to the AR glasses and lack of functionality in the glasses cannot simply be solved and require more resources and collaboration with the manufacturer of the AR equipment.

Continuing research regarding the implementation of AR into the manufacturing
process offers high potential for a case study in addition to real-world applications.
Actively contributing to Industry 4.0 and Smart Manufacturing will continue to grow
the field in addition to further advance and optimize the manufacturing process thus
increasing productivity, efficiency, safety, and saving money. Refinement of the sensor
array enables the opportunity for highly customized data tracking and therefore detailed
feedback to the user. RFID tags are currently used in many industries but are limited in
their tracking capability. The use of AR will allow for tracking throughout the entire
manufacturing process and therefore potentially prevent recalls, prevent full process
shut downs, and potentially save parts that have encountered manufacturing errors. Re-
garding industries that do not have any sort of tracking, a system like this could revo-
lutionize their company. Being able to track the parts your company is producing offers
many benefits and allows a company to utilize the collected data and become more
efficient and productive in their manufacturing.

The utilization of the Arduino based sensor system allows for high customization.
Many off-the-shelf sensors are non-customizable and therefore you either collect all of
the data that they're capable of collecting or you don't use the sensor at all. Our system
allows for the user or company to decide which data they want to collect and which
data is negligible to them. This reduces the data processing time and therefore gives the
user more in-depth feedback in less time.
## Acknowledgements

This work was supported by the J. Wayne & Kathy Richards Faculty Fellowship in Engineering at West Virginia University and the Research Apprenticeship Program (RAP) at the WVU Office of Undergraduate Research.

## References

1. Schlick, J. (2012, May). Cyber-physical systems in factory automation-Towards the 4th industrial revolution. In 2012 9th IEEE International Workshop on Factory Communication Systems (pp. 55-55). IEEE.

2. Ong, S. K., Yuan, M. L., & Nee, A. Y. C. (2008). Augmented reality applications in manufacturing: a survey. International journal of production research, 46(10), 2707-2742.

3. Zheng, P., Sang, Z., Zhong, R. Y., Liu, Y., Liu, C., Mubarok, K., ... & Xu, X. (2018). Smart manufacturing systems for Industry 4.0: Conceptual framework, scenarios, and future perspectives. Frontiers of Mechanical Engineering, 13(2), 137-150.

4. Salonen, A., & Tabikh, M. (2016). Downtime costing—attitudes in swedish manufacturing industry. In Proceedings of the 10th World Congress on Engineering Asset Management (WCEAM 2015) (pp. 539-544). Springer, Cham.

5. Qeshmy, D. E., Makdisi, J., da Silva, E. H. D. R., & Angelis, J. (2019). Managing Human Errors: Augmented Reality systems as a tool in the quality journey. Procedia Manufacturing, 28, 24-30.

6. Vyas, K. (2018, March 12). 5 Cool Augmented Reality Applications in the Manufacturing Sector. Retrieved from https://interestingengineering.com/augmented-reality-applications-manufacturing-sector

7. Rathbone, E. (2018, August 18). Real world applications of Augmented Reality (AR) in manufacturing. Retrieved from http://www.manufacturinglounge.com/real-world-applications-of-augmented-reality-ar-in-manufacturing/

8. Kellner, T. (2018, April 20). How Augmented Reality Improves Manufacturing Productivity. Retrieved from https://www.ge.com/reports/game-augmented-reality-helping-factory-workers-become-productive/

9. Anderl, R. (2014). Industrie 4.0-advanced engineering of smart products and smart production. In Proceedings of international seminar on high technology (Vol. 19).

10. Behzadi, Ajang. (2016). Using Augmented and Virtual Reality Technology in the Construction Industry, 350-351.

11. Mittal, S., Khan, M. A., Romero, D., & Wuest, T. (2019). Smart manufacturing: characteristics, technologies and enabling factors. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture, 233(5), 1342-1361.

12. Nee, A. Y., Ong, S. K., Chryssolouris, G., & Mourtzis, D. (2012). Augmented reality applications in design and manufacturing. CIRP annals, 61(2), 657-679.

13. Uva, A. E., Battullo, M., Manghisi, V. M., Spagnulo, D., Cascella, G. L., & Fiorentino, M. (2018). Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. The International Journal of Advanced Manufacturing Technology, 94(1-4), 509-521.