# Fault Detection in Automatic Manufacturing Processes via 2D Image Analysis Using a Combined CNN-LSTM Model

Na Hyeon Yu, Sujeong Baek

<figure><img src="image_1.png" /></figure>

► To cite this version:

Na Hyeon Yu, Sujeong Baek. Fault Detection in Automatic Manufacturing Processes via 2D Image Analysis Using a Combined CNN-LSTM Model. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.11-18, 10.1007/978-3-031-16407-1_2. hal-05386009

HAL Id: hal-05386009

https://inria.hal.science/hal-05386009v1

Submitted on 27 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Fault Detection in Automatic Manufacturing Processes via 2D Image Analysis Using a Combined CNN-LSTM Model

Na Hyeon Yu and Sujeong Baek

Hanbat National University, Daejeon 34158, Republic of Korea
sbaek@hanbat.ac.kr

**Abstract.** Artificial intelligence, automated sensors, and robots have improved the production efficiency in smart factories. Real-time monitoring and quick responses are important to prevent time losses and increased costs caused by abnormal situations in an automated manufacturing process with several actuators. In this study, we explicitly analyzed only sequential 2D image data from a webcam to detect faults in an automatic manufacturing process without any additional sensors. A model combining a convolutional neural network and long-short term memory was used to reflect the characteristics of sequential processes. When a current process image was applied to the model, it predicted the corresponding future image captured by the camera 1 s later. For training, only image datasets collected during normal manufacturing process operations were used. The prediction error of the training datasets was used for threshold calculation to detect faults. For validation and verification, we collected image datasets for two fault types; the proposed model demonstrated the highest detection accuracy with a dynamic threshold.

**Keywords:** 2D image data · Fault detection · Convolutional neural network · Long short-term memory

## 1 Introduction

Since the 4th industrial revolution, several efforts have been dedicated towards the adoption of automated and intelligent systems in factories in an attempt to develop smart factories. In smart factories, productivity can be improved by combining artificial intelligence and the Internet of Things [1]. In an automated manufacturing process consisting of multiple actuators and sensors, a late response to abnormal conditions may lead to time losses and increased costs and reduce process efficiency. Thus, process monitoring and fault detection are important for efficient operation in an automated manufacturing system [2]. Generally, fault detection in mechanical systems is performed through analysis of analog sensor data, including vibration and temperature signals. Yang *et al.* collected sensor data using a Raspberry Pi microcontroller and created a fault detection algorithm using a convolutional neural network (CNN) model to identify the occurrence
of a fault in an installed sensor in real time [3]. Lim *et al.* developed a fault prediction model by analyzing waveforms of voltage and current signals collected from the power distribution system [4]. The proposed model was found to demonstrate better performance than simple algorithms adopted in the commercial power distribution intelligence system. However, the model required additional sensors, which resulted in corresponding additional costs. In addition, it was also necessary to consider the type, number, and location of the sensors before fault detection model development. For example, Song and Baek conducted real-time fault detection for a sequential manufacturing process using programmable logic controller's digital control signals [5]. They observed a detection delay, but the current situation could not be improved. A possible solution was to install a new sensor at the beginning of the conveyor or analyze the process monitoring images rather than sensor signals.

As mentioned above, in recent years, several studies using 2D images for fault detection and diagnosis have been conducted. Chen *et al.* extracted relevant features by converting the collected sensor signals into the frequency domain for improving the performance of bearing fault diagnosis. After training a CNN, a long-short term memory (LSTM) was used to identify the type of faults, and the approach demonstrated good performance even in the presence of abundant noise [6]. Jalayer *et al.* proposed a contextual LSTM technique that reflected spatio-temporal features for the detection and diagnosis of faults in rotational machinery [7]. The features were extracted by applying fast Fourier transform to the collected data, following which a CNN and an LSTM were used. The approach demonstrated better performance than other basic deep learning models. However, so far, previous studies conducted on fault detection have usually adopted sensor signal transformed images as representative inputs; therefore, dedicated signal preprocessing is still required to generate representative input images.

To skip the elaborate preprocessing step required for input data, in this study, we aimed to detect the current state using only process images captured via a webcam. We used sequential 2D monitoring images captured by a webcam to monitor the process operation status, and we detected faulty situations in an automation process by explicitly analyzing the captured images. We conducted resizing operations as an image preprocessing task. For fault detection, a model combining a CNN and an LSTM was used to consider temporal characteristics of the recorded images and to reflect the characteristics of a sequential process.

## 2 2D Image Data Collection from an Automated Laser-Marking Process

In this study, we collected datasets from a laser-marking process that is installed at Hanbat National University. Typically, the laser-marking process imprints specific letters and a QR code on a product in the production process, inhere a universal serial bus (USB). A pneumatic gripper moves the USB along the target workbench; when a pallet with a USB arrives at a pre-determined position, laser-marking is performed on the USB. To collect 2D monitoring image data, a camera (Logitech C922 PRO HD STREAM WEBCAM) was installed on top of the testbed to capture all movements from the start to the end
of the process. During this period, images were captured at a rate of 1 image/s using Python programming.

During normal process operation, the time elapsed between the entrance of a pallet until the next process is approximately 64 s. If a fault occurs because the pallet enters without a USB or the laser-marking machine is turned off, the process is suddenly terminated at a certain instant, and the operations are halted; note that in our study, for data analysis, 64 sequential images were captured after the process began (the average process time in normal operation). A total of 3,840 images were collected during 30 normal process cycles and 30 abnormal process cycles; 64 images were continuously captured per process cycle, as shown in Table 1. For fault detection, two arbitrary faulty states were generated during the process: (i) a pallet entering without a USB (Fault type I), and (ii) no laser marking due to machine error (Fault type II). These detailed situations are illustrated in Fig. 1. Hereafter, we use “normal data” and “faulty data” to indicate the 2D image datasets collected during normal process operations and abnormal process operations, respectively.

<table><caption>**Table 1.** Number of image datasets collected during normal and abnormal process operations.</caption><thead><tr><th></th><th>Number of process cycles</th><th>Number of images in one cycle</th><th>Total number of images</th></tr></thead><tbody><tr><td>Normal data</td><td>30</td><td>64</td><td>1920</td></tr><tr><td>Faulty data</td><td>30</td><td>64</td><td>1920</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** 2D images collected from laser-marking process in normal and faulty states: (a) Normal data; (b) Faulty data from Fault type I (pallet without USB); (c) Faulty data from Fault type II (no laser-marking operation). Red boxes indicate the difference between normal and abnormal process status at the same time step.</figcaption></figure>
## 3 Fault Detection via a CNN with an LSTM

As the collected datasets could not be easily analyzed via conventional statistical inference, we used an artificial neural network (ANN). In an ANN, high-dimensional input data are planarized in one dimension (as a concatenated vector) and connected to a fully connected layer for classification (normal or fault state). However, as the dimension increases, data loss occurs more easily during planarizing (concatenating multiple rows). In particular, image data are generally composed of three or more dimensions, as two-dimensional RGB images are collected in the form of a time series. Thus, if the existing fully connected layer is identically used with a concatenated vector, spatial information may be lost, rendering fault detection difficult [8]. To effectively detect faults using 2D sequential images, we used a neural network with convolutional layers (CNN) to consider spatial information [9]. CNNs extract significant features while maintaining the input data form for a 2D image; thus, there is no loss in the spatial information, and the output has the same dimensionality as the input data [10]. The data collected in this study were in the form of a time series, which could describe a sequential process over time. To reflect time-variant characteristics, a CNN algorithm combined with an LSTM, instead of a simple CNN, was used [11–13]. In this regard, several studies are currently under way to analyze spatio-temporal characteristics by using a combination model of CNN and LSTM. Huang et al. proposed a CNN–LSTM model that is capable of providing time information on fault occurrences during chemical processes [14]. Jalayer et al. proposed a contextual LSTM technique that identified spatial features through a CNN and temporal features through an LSTM for fault detection in a rotating machine [7].

The model requires only normal data. However, as a supervised learning model, the next $(i + 1)^{th}$ image is always generated as the model output. In our analysis, out of the 64 total images, 63 images (1–63 s) in a specific cycle were designated as the model input, and another 63 images (2–64 s) in the same cycle were designated as the model output. By training the corresponding output data of the $i^{th}$ image as the $(i + 1)^{th}$ image, it was possible to predict the next movement in the process. Finally, using the predicted and actual obtained $(i + 1)^{th}$ images together, we attempted to detect faults in the process with unsupervised learning. We compared the pixel differences between the actual future image and the image predicted by the proposed model. If the difference exceeded a certain threshold, a fault was identified.

Using this detection concept, the final structure of the proposed model was generated, as illustrated in Fig. 2. The current RGB image was resized dimensionality to lower computation costs during training. As mentioned earlier, in the first CNN and LSTM to 18 × 32 to reduce combination layer (ConvLSTM2D layer), 64 filters with a kernel size of applied to the resized image, and the hyperbolic tangent $3 \times 3$ were (tanh) function was used as an activation function. Thereafter, the ConvLSTM2D layer with the same hyper-parameter was repeated thrice, with batch normalization performed at each layer. After all the ConvLSTM2D layers and batch normalizations were completed, a convolution layer with three dimensions (Conv3D) was applied using three filters with a kernel size of 3 × 3 × 3, and the sigmoid function was used as an activation function. Detailed hyper-parameter information is presented in Table 2. A general CNN has a binary classification; the output label is either 0 or 1. Thus, binary cross-entropy is usually used as a loss function [15]. However, the proposed method predicts the next
consecutive image. In other words, the model output consists of continuous values rather than discrete labels. Thus, the mean squared error (MSE), the average squared difference between the predicted values and the actual values [16], was used as a loss function. If the MSE value between the predicted image and the actual future image (collected after 1 s) was greater than the threshold value, it was determined that a fault occurred in the process.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Structure of proposed model (CNN combined with LSTM).</figcaption></figure>

<table><caption>**Table 2.** Optimized hyper-parameter of the proposed CNN model with an LSTM for predicting the next future image.</caption><thead><tr><td>Hyper-parameter</td><td>Value</td></tr></thead><tbody><tr><td>Optimizer function</td><td>Adam</td></tr><tr><td>Loss equation</td><td>Mean squared error</td></tr><tr><td>Batch size</td><td>8</td></tr><tr><td>Number of epochs</td><td>400</td></tr></tbody></table>

## 4 Fault Detection Results for the CNN Algorithm with an LSTM

A threshold value for fault detection must be determined during the training phase. In this study, we employed two thresholding methods: (i) a fixed threshold and (ii) a dynamic threshold. The fixed threshold is a method in which all the trained datasets are correctly classified as normal datasets based on the maximum value of the MSE in the datasets obtained through normal processes (hereafter referred to as normal datasets). On the contrary, the dynamic threshold method proceeds by considering the characteristics of each time zone [17]. A dynamic threshold consists of *n*-1 values (63 values in this paper), which are the summation of the average value and six times the standard deviation of the MSE values for each time zone. Each calculation result is considered as the threshold at each time point.

Based on this concept, 30 normal datasets were used to develop the trained model; additionally, 30 datasets (hereafter called as faulty datasets) were further collected to analyze the fault detection performance. Figure 1 depicts faulty situations encountered
in obtaining the faulty datasets. To test the performance of the proposed model, the model was compared to a basic LSTM model. On applying the fixed threshold method to the basic LSTM model, it resulted in 14 fault occurrence alarms for 30 faulty datasets. However, all the fault alarms were identified as false alarms, as shown in Table 3. The proposed CNN-LSTM model with a fixed threshold demonstrated better detection results: 8 faults were correctly detected among 30 detection alarms (for every fault situation). Finally, the proposed fault detection model with a dynamic threshold demonstrated the best detection results (27 out of 30 fault occurrence and 3 false alarms).

<table><caption>**Table 3.** Fault detection accuracy according to fault detection methods (analyzing test datasets).</caption><thead><tr><th></th><th>A basic LSTM with a fixed threshold</th><th>The proposed CNN-LSTM with a fixed threshold</th><th>The proposed CNN-LSTM with a dynamic threshold</th></tr></thead><tbody><tr><th>Number of true positives</th><td>0/30</td><td>8/30</td><td>27/30</td></tr></tbody></table>

For a more detailed analysis of the detection results, we monitored the detection time. The fault detection time for the best fault detection model (CNN-LSTM with the dynamic threshold) is also summarized in Table 4, according to the fault types. In the case of Fault Type I, a pallet entered without a USB, an actual fault occurred within 3.6 s on average, and the model detected the corresponding fault occurrences within approximately 3.5 s, which is almost identical to the actual fault occurrence time. In the case of Fault Type II, a process error occurred due to a fault in the laser-marking operation, an actual fault occurred within 38.5 s on average, and the proposed model detected the fault occurrences within 40 s, which indicated a slight delay. The obtained detection results appear satisfactory; however, the model demonstrates a limitation in predicting fault occurrences: it can judge only the current situation. If a fault is detected by the proposed model before it actually occurs (for example, more than 2 s early), it would be highly probable of an erroneous detection (false alarm). In addition, as the faults considered in this study do not result from deterioration or machine aging, they cannot be predicted in advance (fairly early). To address this limitation, information recorded for the purpose of manufacturing management (such as operation time of a cylinder or conveyor) should be collected using given images and analyzed using target values.

<table><caption>**Table 4.** Time information for actual fault occurrence and detection by the proposed model (CNN-LSTM with a dynamic threshold) for two fault types.</caption><thead><tr><th></th><th>Average time of actual fault occurrence</th><th>Average time of fault detection</th></tr></thead><tbody><tr><th>Fault type I</th><td>3.6</td><td>3.5</td></tr><tr><th>Fault type II</th><td>38.5</td><td>40.0</td></tr></tbody></table>
# 5 Conclusion

In this study, a fault detection model was developed for an automated manufacturing process; 2D images were analyzed instead of the addition of another analog sensor. To consider the characteristics of sequential 2D image data captured by a webcam, fault detection was performed using a CNN algorithm combined with an LSTM. The algorithm predicted the next image (in this paper, 1s later) by analyzing the current image. Using the predicted image, faults were detected by comparing pixel differences between the predicted and actual images. To improve the detection performance, two different thresholding methods were employed: a fixed threshold and a dynamic threshold. To compare the performance, three different detection models were analyzed: a basic LSTM with a fixed threshold, the proposed CNN-LSTM with a fixed threshold, and the proposed CNN-LSTM with a dynamic threshold. In terms of the detection rate and detection time, CNN-LSTM with a dynamic threshold demonstrated the best performance. In summary, because the proposed method employs only 2D sequential images recorded using a camera, it can be easily applied to any discrete manufacturing process, provided that monitoring images can be captured using a camera. In addition, we attempted to exclude detailed preprocessing steps for better utilization.

However, there is still further scope for improvement in the detection performance. For example, in the proposed model, equally weighted training for each pixel was conducted, but each pixel might have different attention, for example more attention to an actuator's significant movement and less attention to background. In future, the fault detection performance can also be improved by analyzing significant areas in the captured images. To effectively diagnose and repair a detected fault, various bits of information, such as the type of faults and corresponding repair methods, can be automatically derived better depending on the fault detection results.

**Acknowledgement.** This work was supported by project for Industry-Academic Cooperation Based Platform R&D funded Korea Ministry of SMEs and Startups in 2020 (Project No. S3025721), and supported by project for Industry-Academic Cooperation Based Platform R&D funded Korea Ministry of SMEs and Startups in 2020 (Project No. S3025825).

This paper was also supported by Korea Institute for Advancement of Technology (KIAT) grant funded by the Korea Government (MOTIE). (P0012744, HRD program for industrial innovation).

# References

1. Seo, I.B., Cho, Y.B.: Comparison and analysis of chest X-ray-based deep learning loss function performance. J. Korea Inst. Info. Comm. Eng. **25**(8), 1046–1052 (2021)

2. Song, G.W., Kim, B.S., Choi, W.S.: RCM based failure-prediction system for equipment. In: Proceedings of 2010 Fall Korean Society of Mechanical Engineers, pp. 1281–1286. The Korean Society Mechanical Engineering, Republic of Korea (2010)

3. Yang, J.W., Lee, Y.D., Koo, I.S.: Timely sensor fault detection scheme based on deep learning. J. Inst. Internet, Broadcast. Comm. **20**(1), 163–169 (2020)

4. Lim, H.G., Han, K.H., Kim, Y.S., Kim, J.H.: Algorithm for detection of fault through analysis of waveform from voltage and current in distribution system. In: Proceedings of 2016 Fall Korean Institute of Electrical Engineers Conference, pp. 101–104. Korean Institute of Electrical Engineers, Republic of Korea (2016)
5. Song, Y.U., Baek, S.J.: Real-time fault detection in discrete manufacturing systems via LSTM model based on PLC digital control signals. J. Korean Soc. Ind. Syst. Eng. **44**(2), 115–123 (2021)

6. Chen, X., Zhang, B., Gao, D.: Bearing fault diagnosis base on multi-scale CNN and LSTM model. J. Intell. Manuf. **32**(4), 971–987 (2020). https://doi.org/10.1007/s10845-020-01600-2

7. Jalayer, M., Orsenigo, C., Vercellis, C.: Fault detection and diagnosis for rotating machinery: a model based on convolutional LSTM, fast Fourier and continuous wavelet transforms. Comput. Ind. **125**, 103378 (2021)

8. Kim, Y.S., Kim, N.H., Kim, J.H.: Hardware architecture of artificial neural network for large-scale network applications, In: Proceedings of 2015 Summer Annual Conference of Institute of Electronics and Information Engineers, pp. 72–75. The Institute of Electronics and Information Engineers, Republic of Korea (2015)

9. Kang, H.J.: SoftMax computation in CNN using input maximum value. J. Korea Inst. Inf. Comm. Eng. **26**(2), 325–328 (2022)

10. Lim, J.Y., Kim, D.H., Noh, T.W., Lee, B.K.: Remaining useful life prediction for lithium-ion batteries using EMD-CNN-LSTM hybrid method. Trans. Korean Inst. Power Electr. **27**(1), 48–55 (2021)

11. Lee, H.H., Hong, N.Y., Lee, T.W.: Comparison of prediction for multivariate time series between LSTM and VARX. J. Korean Data Inf. Sci. Soc. **33**(1), 35–47 (2022)

12. Kim, K.T., Lee, B.M., Kim, J.W.: Feasibility of deep learning algorithms for binary classification problems. J. Korean Intell. Inf. Syst. Soc. **23**(1), 95–108 (2017)

13. Cho, T.S., Lee, M.H., Kim, M.S., Song, B.H., Park, P.K.: Heating furnace temperature control system of anomaly detection method using LSTM autoencoder, In: Proceedings of 2021 Korean Institute of Electrical Engineers Conference, pp. 269–270. Korean Institute of Electrical Engineers, Republic of Korea (2021)

14. Huang, T., Zhang, Q., Tang, X., Zhao, S., Lu, X.: A novel fault diagnosis method based on CNN and LSTM and its application in fault diagnosis for complex systems. Artif. Intell. Rev. **55**, 1289–1315 (2022)

15. Kim, Y.H., Woo, G.M. Kim, H.B., Park, H.H.: Life pattern-based depression prediction machine learning method using NHANES dataset, In: Proceedings of the Korean Institute of Communication Sciences Conference, pp. 81–82. Korean Institute of Communication Sciences, Republic of Korea (2021)

16. Son, S.J., Chen, T.J., Park, A.R., Baek, S.J.: Spectrum data compression using ResNet-convolutional autoencoder based neural network. J. Korea Academia-Ind. Coop. Soc. **22**(12), 135–143 (2021)

17. Yu, N.H., Baek, S.: Improving next image prediction performance using dynamic threshold for deep learning based fault detection in automated systems. In: Proceedings of the Korean Society of Mechanical Engineers Conference, p. 32. The Korean Society of Mechanical Engineers, Republic of Korea (2022)