<figure><img src="image_1.png" /></figure>

HAL
open science

# Detection of Fluid Level in Bores for Batch Size One Assembly Automation Using Convolutional Neural Network

Alexej Simeth, Jessica Plaßmann, Peter Plapper

► To cite this version:

Alexej Simeth, Jessica Plaßmann, Peter Plapper. Detection of Fluid Level in Bores for Batch Size One Assembly Automation Using Convolutional Neural Network. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.86-93, 10.1007/978-3-030-85906-0_10. hal-04022155

HAL Id: hal-04022155

https://inria.hal.science/hal-04022155

Submitted on 9 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Detection of fluid level in bores for batch size one assembly automation using convolutional neural network

Alexej Simeth¹[0000-0001-7463-5525], Jessica Plaßmann², and Peter Plapper¹[0000-0003-3507-1397]

¹ University of Luxembourg, 6, rue Richard Coudenhove-Kalergi, L-1359, Luxembourg

² Hochschule Trier, Schneidershof, 54293 Trier, Germany

alexej.simeth@uni.lu

**Abstract.** Increased customization and shortening product life cycles pose a challenge for automation, especially in assembly. In combination with the nature of assembly tasks, which may require high level of perception, skill, and logical thinking, these tasks are often conducted manually, especially in certain industries (e.g. furniture, power tools) or small and medium-sized enterprises. One of such tasks is the liquid level monitoring in gluing processes. Existing non-manual solutions are based on conventional and less flexible algorithms to detect the current liquid level. In production environments with highly individualized products, a need for more performant models arises. With artificial intelligence (AI) it is possible to deduct decisions from unknown multidimensional correlations in sensor data, which is a key enabler for assembly automation for products with high degree of customization.

In this paper, an AI-based model is proposed to automate a gluing process in a final assembly. Images of a gluing process are taken with a camera and a convolutional neural network is used to extract images features. The features are applied to train a support vector machine classifier to identify the liquid level. The developed model is tested and validated with a Monte-Carlo-simulation and used on a demonstrator to automate a gluing process. The developed model classifies images of liquid levels with over 98% accuracy. Similar results are achieved on the demonstrator.

**Keywords:** liquid detection, artificial intelligence, convolutional neural network, camera, assembly, automation

## 1 Introduction

Assembly processes are still the part of manufacturing with a low overall degree of automation [1]. This is due to the nature of assembly tasks, which often require a high level of perception, skill, and logical thinking [2]. Shorter product life cycles and higher customization lead to increased variance in product portfolio with low volumes up to batch size one. Since every product is going through final assembly, complete variance must be covered, which is further inhibiting automation. New trends such as industry 4.0 and the rise of artificial intelligence (AI) in manufacturing promote increased automation and are key enabler for intelligent and smart manufacturing systems [3], [4].
For a flexible automation of assembly processes of highly individual products, it is
necessary to replace observations taken by humans with digital systems to start, exe-
cute, or stop (sub-) processes. For large batch sizes, tools, positions, process parameters
etc. are usually adapted to fit a certain product. This procedure is not economical for
small volumes with high product variability [5]. In some cases, process parameters are
not known a-priori due to the characteristics of processed workpieces or the production
environment. Especially in such situations manual work is still necessary. With AI it is
possible to deduct decisions and actions from unknown multi-dimensional correlations
in sensor data [6], which can be used for automation of highly individualized products.

The focus of the authors' research is assembly automation consisting of pick-and-place and gluing processes in batch size one assembly scenarios. Workpieces are joined by placing them into bores in a carrier workpiece and then bonded with glue. Due to the properties of the carrier workpiece, the required amount of glue is unknown and standard volumetric process control is not applicable. In this paper, a model is proposed to detect the fluid level of glue in workpieces so that the mentioned gluing process can be started, monitored, and stopped. An optical sensor is used, which is applicable for different sensing tasks in the overall assembly processes.

The next chapter gives a brief overview of related work and approaches of other authors. The methodology applied in this paper, experimental setup, results, and a conclusion are given in chapters 3-6.

## 2 Related Work

Detection of specific objects, features, etc. is a common issue in manufacturing and automation, and there exist many different solutions for different problems. In [7] the authors describe an image processing system directly related to glue detection, which shall be used in an LCD panel production. The authors showcase a camera-based system which monitors and measures the width of an applied glue line by identifying the edges of the glue line with standard Canny edge detection algorithm and by calculating the distance between edge pixels of two detected edges (both sides of glue line).

Several solutions for liquid / fluid detection in production environment using optical methods are proposed in the area of bottle filling. The camera is positioned so that a side view of the bottle is captured [8]–[12]. In [8], the authors propose to classify a bottle into under-filled, filled, and over-filled by using Canny edge detection algorithm to identify the surface of the liquid level. The detected surface is compared to a pre-defined reference line. The average vertical distance of the detected edge pixels to the reference is taken to classify the bottle. Similarly, an ISEF edge detection filter is applied in [9] to identify the surface of the liquid and the lower edge of the cap. The averaged vertical distance between edge pixels in a defined region of interest (ROI) is compared to a threshold. The proposed method in [10] uses image segmentation based on color. After changing the color space, the extracted image is smoothened, binarized, and the dark liquid is separated from the bright background. The contour of the dark area is taken to calculate the filling level. In [11], a fast and simple method is developed to automatically measure the volume of liquid and the bubble phase on top of the liquid
in translucent cylindric vessels. Key of this method is the installation of an area light
source and a black stripe combined with cropping and merging several image patches,
which emphasize the liquid surface and bubble area. The actual detection is based on
characteristic changes in histogram. In [12] the authors compare a conventional liquid
detection approach via several mean filters with a neural network-based approach. De-
spite the very simple structure of the neural network (three layers), classification results
are slightly improved compared to conventional approach. In [13] a dispensed glue drop
on a workpiece is monitored to detect defects in the glue dispensing system. The authors
use principal component analysis (PCA) to detect variation in the output and state
whether a fault in the system exists.

Showcased methods majorly apply conventional models to detect the surface of the
liquid and compare it against predefined references or thresholds. This requires a con-
stant environment and specific settings for each product. In a continuous changing pro-
duction environment (e.g., robot mounted systems) with always changing products,
more flexible and robust solutions are required.

3 Methodology

In this paper, an AI-based model is developed to robustly detect a fluid on a workpiece in order to automate an industrial gluing process. The development of the model follows the workflow presented in [6], which is depicted in **Fig. 1**. At first, the initial data set is generated. The classification is reduced to a binary problem and single images are labelled as “empty” or “full”. To achieve a sufficient size of data set, the data which is expected to be more influential, i.e., data points close to label change, are augmented following the method of importance sampling [14].

<figure><img src="image_4.png" /><figcaption>Fig. 1. Workflow for supervised and unsupervised Learning [6], p. 22</figcaption></figure>

The data is then randomly split into test and training set. The latter is used to train the
model and to extract image features, which are used for classification. To extract the
image features, a pre-trained convolutional neural network (CNN) is applied, because
it is usually much faster and simpler than designing a new CNN [15]. The extracted
features are used in a machine learning algorithm to classify the test set. Machine learn-
ing algorithms achieve similar performance and accuracy compared to deep learning
classifiers by reducing the computational effort [16]. In the proposed model the 50 layer
deep CNN ResNet50 in combination with a classifier based on a support vector
machine (SVM) are applied. The learned interdependencies are highly depending on
the selected training set, which depends on the random drawing of the initial data set.
An additional cross-validation is conducted. Via a Monte-Carlo-Simulation (MCS) [17]
the impact of the randomly selected training data on the outcome is analyzed.

In a final step, the model is used to classify new images, which are captured during
a live gluing process. The gluing process is automatically stopped based on the result
of the image classification.

# 4 Experimental setup

## 4.1 Applied detection model

The detection of the glue is achieved with a hybrid detection model consisting of a pretrained CNN and an SVM. In a first step image features are extracted by the CNN, i.e. the activations of the output layer of the CNN. The pretrained CNN is not changed during feature extraction. Secondly, the activations of all images of the training set and their labels are used to train the SVM image classifier. The trained classifier is tested and validated with the test set. Here, image features are again extracted by the same pretrained CNN and forwarded to the SVM classifier for classification.

The introduced models of other researches in chapter 2 mostly rely on conventional
filters combined with defined thresholds. The decision rules to classify an image are
programmed manually and applicable only for the specific boundary conditions in each
use case. In contrast, the proposed model applies a decision rule generated by the
computer itself based on the training data resulting in the SVM image classifier. By
changing the ground data set, the model is applicable similarly in other use cases.
Additionally, the resulting SVM classifier is expected to be significantly more robust
towards changes in boundary conditions than conventional methods.

<figure><img src="image_5.png" /><figcaption>Fig. 2. a): Designed test stand for initial data set generation. b): Demonstrator.</figcaption></figure>

## 4.2 Generation of Data Set

In order to separate the modelling from the production line, a test stand is designed to
create an initial data set. Based on this set, the CNN and the classifier are trained for
later use in a real gluing process. The initial data is created by recoding manually conducted gluing trials as depicted in Fig. 2 a). Glue is filled through a nozzle and an inlet hole into a workpiece. A cradle is mounted above the workpiece and a smartphone is used to record the gluing process. The region of interest (ROI), which is the outlet hole, is cropped out of each frame resulting in 41x41 resolution images. The data is transformed into 244x244x3 RGB pixel images, which are the required size for the Res-Net50 input layer. The obtained images are then labelled into the categories (labels) “empty” and “full”. In total 605 images were created and used for further data set optimization. In row “Label” in Table 1 examples for the images of the two labels are given.

## 4.3 Optimization of Data Set

For the training of the model sufficient amount of data is required. To increase the initial data set size, the data set is augmented by rotation and mirroring of selected images following the method of importance sampling. In this case, all the images which are labelled as “empty” but close to full and vice versa are augmented, since it is expected, that these images have a high impact on the decision rule to be determined. The final distribution of the data set is characterized by **Table 1**. Based on the initial 605 video frames, a total data set of 3,000 images is created with importance sampling method.

<table><caption>**Table 1.** Generated and optimized data set for image classification</caption><thead><tr><th>Data</th><th colspan="4">Frames from videos taken on test stand</th></tr><tr><th>Label</th><th colspan="2">Empty (380 images)</th><th colspan="2">Full (225 images)</th></tr><tr><th>Sublabel</th><th colspan="4">Divide into more and less influential data for "importance sampling"</th></tr><tr><th>Data Set</th><th>Clearly empty</th><th>Close to full</th><th>Sufficiently full</th><th>Overfull</th></tr></thead><tbody><tr><td></td><td><img src="empty_data.png" alt="A black circle with a black border and a white circle"></td><td><img src="full_data.png" alt="A white circle with a black border and a black circle"></td><td><img src="full_data.png" alt="A white circle with a black border and a white circle"></td><td><img src="full_data.png" alt="A white circle with a black border and a white circle"></td></tr><tr><td></td><td></td><td><img src="augment_data.png" alt="A black circle with a black border and a white circle"></td><td><img src="augment_data.png" alt="A black circle with a black border and a white circle"></td><td><img src="augment_data.png" alt="A black circle with a black border and a white circle"></td></tr><tr><td></td><td>1830 images</td><td></td><td>1170 images</td><td></td></tr></tbody></table>

## 4.4 Validation

From the label “empty” 1170 images are randomly selected so that both classes are of same size. The generated data set is randomly split into training data and test data in the ratio 70:30. The training set is used to train the model. The test set is classified, and the result is compared to the original label. In an MCS, the training of the SVM image classifier is repeated 500 times with a different random selection of label “empty”, training set, and test set.

In a final validation step, the SVM trained with the identified image features extracted by CNN is used on a technology demonstrator to classify new images, which are not part of the initial data set. On the demonstrator glue is automatically pumped
through a nozzle into the inlet hole and the outlet hole is monitored by an industrial fixed lens camera (see **Fig. 2 b**)). A ring light with red LED is used to illuminate the scenery and to reduce the environmental impact. The red image plane is taken for further processing. The cropped ROI has a different resolution compared to the initial data set and is grayscale but covers a similar physical area. The resulting grayscale image is transformed into the required model input size and classified by the SVM. If an image is classified as full, the gluing process is stopped. The trials are conducted in bright and dark environmental situations.

## 5 Results

The proposed model to detect fluid level in workpieces achieves an overall prediction accuracy of >98%. The model is cross validated with an MCS and trained 500 times with different, randomly selected training and test set configurations. The prediction accuracy varies over the 500 simulations in the interval from 93-99.5% (see **Fig. 3**). All wrong classifications are on images which are directly at the boarder to the other label. I.e., the images are either of the first frames of a video, which are labelled as full or the last frames of a video which are labelled as empty (cf. categories **Table 1**).

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Results of overall classifier performance under varying data based on MCS.</figcaption></figure>

The model is then applied on a technology demonstrator. It is used to stop a gluing process based on the classification of the glue level. At 35 conducted gluing trials, the proposed method stopped the process in all cases correctly. A selection of the images classified as full during the trials is shown in **Table 2** to highlight the differences between the images used for training and the images obtained during the live gluing process. As depicted, the trials are conducted in a daylight and night scenario.

<table><caption>**Table 2.** Images classified as full during demonstrator validation.</caption><tr><td>Bright environment</td><td><img src="image_0.png" alt="Image of a bright environment"></td><td><img src="image_1.png" alt="Image of a bright environment"></td><td><img src="image_2.png" alt="Image of a bright environment"></td><td><img src="image_3.png" alt="Image of a bright environment"></td><td><img src="image_4.png" alt="Image of a bright environment"></td></tr><tr><td>Dark environment</td><td><img src="image_5.png" alt="Image of a dark environment"></td><td><img src="image_6.png" alt="Image of a dark environment"></td><td><img src="image_7.png" alt="Image of a dark environment"></td><td><img src="image_8.png" alt="Image of a dark environment"></td><td><img src="image_9.png" alt="Image of a dark environment"></td></tr></table>
# 6 Summary

In this paper, the authors propose a new model to detect glue level in workpieces. The hybrid detection model consists of the pretrained convolutional neural network ResNet50 and a support vector machine image classifier. The used ground data set is generated on a test stand with a standard smartphone camera where images of a gluing process are taken. The obtained images are labelled into the two classes ‘full’ and ‘empty’. Based on the generated data set, the classifier is trained with image features extracted by the neural network and used to classify new images.

The developed model classifies the test images with a very high accuracy. Based on a cross validation with 500 random distributions of training and test data of the initial data set, in average 98% of all images are classified correctly. Furthermore, the proposed model can be applied in a real gluing process with significantly different boundary conditions compared to the test stand, especially the lighting conditions. In conducted trials on a demonstrator, the developed model stopped a gluing process based on the glue level classification in all cases correctly. It is shown that the proposed AI-based model can deduct decisions from unknown multidimensional sensor data. This is necessary for automation of processes, where important production parameters are not known a-priori. In combination with an automated region-of-interest prediction, which is subject to adjacent research of the authors, a flexible gluing system as part of an assembly system for products with high degree of customization up to batch size one is developed. In ongoing experiments, the developed model is compared to other approaches (conventional filter, machine learning, deep learning) from both computer science and production perspective.

# References

[1] M. Kleindienst and C. Ramsauer, "Der Beitrag von Lernfabriken zu Industrie 4.0-Ein Baustein zur vierten industriellen Revolution bei kleinen und mittelständischen Unternehmen," *Industrie-Management*, vol. 3, pp. 41–44, 2015.

[2] M. Scholer, "Wandlungsfähige und angepasste Automation in der Automobilmontage mittels durchgängigem modularem Engineering - Am Beispiel der Mensch-Roboter-Kooperation in der Unterbodenmontage," Universität des Saarlandes, 2018.

[3] S. S. V. K. Kolla, A. Sanchez, M. Minoufekr, and P. Plapper, "Augmented Reality in Manual Assembly Processes," *Augment. Real. Man. Assem. Process.*, pp. 121–128, 2020.

[4] L. Wang, "From Intelligence Science to Intelligent Manufacturing," *Engineering*, vol. 5, no. 4, pp. 615–618, Aug. 2019, doi: 10.1016/j.eng.2019.04.011.

[5] P. Burggräf, J. Wagner, M. Dannapfel, S. Fluchs, K. Müller, and B. Koke, "Automation decisions in flow-line assembly systems based on a cost-benefit analysis," *Procedia CIRP*, vol. 81, pp. 529–534, 2019, doi: 10.1016/j.procir.2019.03.150.

[6] R. Akerkar, *Artificial Intelligence for Business*. Cham: Springer International Publishing, 2019.

[7] Yuchung Huang, Chiahsiang Wu, and Chih-Yuan Chang, "An application of image
processing in flat panels,” in *2012 International Conference on Wavelet Active Media Technology and Information Processing (ICWAMTIP)*, Dec. 2012, pp. 8–11, doi: 10.1109/ICWAMTIP.2012.6413427.

[8] M. A. A. Felipe, T. V Olegario, N. T. Bugtai, and R. G. Baldovino, “Vision-based Liquid Level Detection in Amber Glass Bottles using OpenCV,” in *2019 7th International Conference on Robot Intelligence Technology and Applications (RiTA)*, Nov. 2019, pp. 148–152, doi: 10.1109/RITAPP.2019.8932807.

[9] K. J. Pithadiya, C. K. Modi, and J. D. Chauhan, “Machine vision based liquid level inspection system using ISEF edge detection technique,” in *Proceedings of the International Conference and Workshop on Emerging Trends in Technology*, Feb. 2010, pp. 601–605, doi: 10.1145/1741906.1742044.

[10] M. M. Gonzalez Ramirez, J. C. Villamizar Rincon, and J. F. Lopez Parada, “Liquid level control of Coca-Cola bottles using an automated system,” in *2014 International Conference on Electronics, Communications and Computers (CONIELECOMP)*, Feb. 2014, pp. 148–154, doi: 10.1109/CONIELECOMP.2014.6808582.

[11] H. Ma and L. Peng, “Vision Based Liquid Level Detection and Bubble Area Segmentation in Liquor Distillation,” in *2019 IEEE International Conference on Imaging Systems and Techniques (IST)*, Dec. 2019, pp. 1–6, doi: 10.1109/IST48021.2019.9010097.

[12] T. Beck, B. Gatternig, and A. Delgado, “Schaum-und Füllstanderkennung mittels optischer Systeme mit neuronalen Algorithmen,” in *Fachtagung „Experimentelle Strömungsmechanik“*, 2019, pp. 17.1-17.7.

[13] Z. Huang, V. C. Angadi, M. Danishvar, A. Mousavi, and M. Li, “Zero Defect Manufacturing of Microsemiconductors – An Application of Machine Learning and Artificial Intelligence,” in *2018 5th International Conference on Systems and Informatics (ICSAI)*, Nov. 2018, pp. 449–454, doi: 10.1109/ICSAI.2018.8599292.

[14] B. Arouna, “Adaptative Monte Carlo Method, A Variance Reduction Technique,” *Monte Carlo Methods Appl.*, vol. 10, no. 1, pp. 1–24, Jan. 2004, doi: 10.1515/156939604323091180.

[15] MathWorks, “Pretrained Deep Neural Networks,” 2021. https://www.mathworks.com/help/deeplearning/ug/pretrained-convolutional-neural-networks.html (accessed Apr. 08, 2021).

[16] A. Kumar T.K., R. Vinayakumar, S. Variyar V.V., V. Sowmya, and K. P. Soman, “Convolutional Neural Networks for Fingerprint Liveness Detection System,” in *2019 International Conference on Intelligent Computing and Control Systems (ICCS)*, May 2019, pp. 243–246, doi: 10.1109/ICCS45141.2019.9065713.

[17] C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan, “An introduction to MCMC for machine learning,” *Mach. Learn.*, vol. 50, no. 1, pp. 5–43, 2003, doi: 10.1023/A:1020281327116.