# Process monitoring by deep neural networks in directed energy deposition: CNN-based detection, segmentation, and statistical analysis of melt pools

Reza Asadi, Antoine Queguineur, Olli Wiikinkoski, Hossein Mokhtarian, Tommi Aihkisalo, Alejandro Revuelta, Iñigo Flores Ituarte

<figure><img src="image_1.png" /></figure>

## To cite this version:

Reza Asadi, Antoine Queguineur, Olli Wiikinkoski, Hossein Mokhtarian, Tommi Aihkisalo, et al. Process monitoring by deep neural networks in directed energy deposition: CNN-based detection, segmentation, and statistical analysis of melt pools. Robotics and Computer-Integrated Manufacturing, 2024, 87 (7), pp.102710. 10.1016/j.rcim.2023.102710. hal-04790717

HAL Id: hal-04790717

https://hal.science/hal-04790717v1

Submitted on 22 May 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

Check for
updates

# Process monitoring by deep neural networks in directed energy deposition: CNN-based detection, segmentation, and statistical analysis of melt pools

Reza Asadi<sup>a, *</sup>, Antoine Queguineur<sup>a, b</sup>, Olli Wiikinkoski<sup>a</sup>, Hossein Mokhtarian<sup>a</sup>, Tommi Aihkisalo<sup>c</sup>, Alejandro Revuelta<sup>c</sup>, Íñigo Flores Ituarte<sup>a</sup>

## ARTICLE INFO

**Keywords:**

Additive manufacturing

Direct energy deposition

Artificial intelligence

Melt pool monitorings

CNN-based segmentation

Statistical analysis

## ABSTRACT

The complex interaction between laser and material in Laser Wire Direct Energy Deposition (LW-DED) Additive Manufacturing (AM) benefits from process monitoring methods to ensure process stability and final part quality. Understanding the relationship between process parameters and melt pool geometrical characteristics can be used to effectively monitor and in-process control the process, as the melt pool geometrical characteristics serve as crucial indicators of process stability and quality.

This study presents a novel in-situ monitoring approach for LW-DED, utilizing process images for melt pool segmentation, melt pool geometrical characteristics estimation, process stability assessment, and bead geometry prediction. The segmentation of melt pool objects was successfully accomplished using Convolutional Neural Networks (CNN)-based models, enabling the prediction of essential parameters such as melt pool area, height, width, center of area, and the center point of the bounding box enclosing the melt pool. Multiple models were compared regarding the accuracy and processing speed using a controlled central composite design and random experiments. We used an Inconel alloy 625 wire and two distinct substrate materials for deposition, a coaxial laser welding head with a 3 kW fiber laser, and an off-axis welding camera for monitoring.

Among the CNN architectures evaluated, YOLOv8l demonstrated superior accuracy with mean Average Precision (mAP) values of 0.925 and 0.853 for Stainless Steel (SS) and low carbon steel (S355) substrates, respectively. Additionally, YOLOv8s exhibited a notable processing speed of over 114 frames per second, which indicates its suitability for real-time process control. Furthermore, the results indicate a significant correlation between process parameters and melt pool geometry variables. Notably, a clear correlation was established between melt pool characteristics and bead geometries obtained through microscopic examinations, including penetration depth and heat-affected zone. The analysis revealed a significant correlation for the bead area and width parameters. In relation to the bead height, while the correlation exhibited a lower magnitude compared to bead area and width, it remained responsive. In addition, the tensor masks derived from the developed models have proven to be highly effective in accurately predicting bead geometries.

The results demonstrate the effectiveness of YOLO-based algorithms for detecting and segmenting the melt pool. Statistical analysis confirms the significance of stabilized process data and the accuracy of melt pool geometric models. We demonstrate that integrating advanced monitoring and control techniques using artificial intelligence methods like CNN can facilitate process stability and quality control.

## 1. Introduction

Additive Manufacturing (AM) has revolutionized metal part production, presenting a powerful alternative approach to traditional manufacturing processes such as milling or casting [1–4]. Direct Energy

Deposition (DED) emerges as a highly viable option for large-scale AM applications, with its potential extending across numerous industrial processes [5–8]. DED is an additive manufacturing technique that employs focused energy mainly powered by a source such as a laser beam, electron beam, and plasma arc to deposit material onto a substrate,

<footnote><sup>a</sup> Faculty of Engineering and Natural Sciences, Tampere University, Korkeakoulunkatu 6, 33014, Tampere, Finland</footnote>

<footnote><sup>b</sup> Ecole Centrale Nantes, GeM-UMsR CNRS 6183, 1 rue de la Noé, 44321, Nantes, France</footnote>

<footnote><sup>c</sup> Technical Research Centre of Finland Ltd, P.O. Box 1000, FI-02044, VTT, Finland</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: reza.asadi@tuni.fi (R. Asadi).</footnote>
facilitating the production of intricate 3D structures [9,10]. Laser-based DED (L-DED) employs a laser beam to generate a small melt pool on a substrate material's surface and is utilized to produce precision components demanding higher dimensional accuracy [11]. In L-DED, the laser melts filler material in powder or wire and adds it to the melt pool. Laser Wire-Deposition (LW-DED), a notable variant, capitalizes on wire-based processes, combining the benefits of DED and wire-based methodologies. LW-DED finds extensive applications in areas such as repair, cladding, and near-net shape part manufacturing [12].

The wire-based filler material in LW-DED processes offers several advantages over powder, including (i) higher deposition rates (i.e., up to 40 lb/h (18.1437 kg/h) for titanium alloy, while the powder-fed deposition system achieves a lower rate of 1.6 lb/h (0.7257 kg/h) [9]), (ii) enhanced repeatability levels, attributed to the simplified setup and operation of the process, (iii) lower material costs, (iv) superior material utilization, achieving levels close to 100 %, and (v) enhanced surface quality of the deposited material, resulting in superior surface characteristics [12]. In addition, resistance heating of the wire in LW-DED processes has been proven to enhance melting rates and process stability. On the other hand, LW-DED processes have limitations to consider: restricted material availability in wire form compared to Laser Powder DED, challenges in achieving precise details and intricate features in complex geometries due to factors like wire diameter and deposition strategies, the need for precise control and monitoring systems for consistent wire feeding, and potential heat-related issues from the high laser energy input [13].

This study examines LW-DED processes where the laser source focuses on the substrate to melt the wire and heat the substrate while utilizing Argon gas. In laser-wire DED, the wire is introduced into the process either laterally or coaxially. The process heads commonly used for powder-DED, which utilize coaxial laser beam guidance, can also be adapted for LW-DED systems with lateral wire feed [12]. The directional dependence of lateral wire feeder in LW-DED processes limits its 3D capability, making it challenging to effectively coat and 3D print complex surfaces [14]. A coaxial feeding wire is employed to enhance the performance of LW-DED processes by mitigating the directional dependence. By guiding the wire through the center of the process head, laser beams can be precisely focused on a common working point [14].

The rapid solidification of the melt pool in L-DED processes creates temperature gradients, leading to residual stresses, warping, and defects. These issues affect the local microstructure and mechanical properties, highlighting the necessity for online monitoring and control in such processes [15]. Besides, precise adjustment of controllable process parameters such as Wire Feed Rate (WFR), Travel Speed (TS), and Laser Power (LP), and their impact on the melt pool characteristics is essential for determining the printed part geometries and surface characteristics [16].

Experimental findings highlight the requirement for an in-situ monitoring and control system in LW-DED processes. Formation of non-uniform build morphology, including bead width increase, uneven bead height, and increased melt pool depth, necessitates monitoring systems [11,17]. Considering these factors, researchers have actively advanced sensing, modeling, monitoring, and control techniques for DED, particularly L-DED processes.

The literature review encompassed various measurement systems that capture melt pool and temperature information, such as thermocouples, pyrometers, and cameras. The existing studies on DED process monitoring and control primarily concentrate on assessing melt pool geometry, deposition geometry, and thermal management aspects [17, 18]. In [19], a coaxial monitoring system containing a custom laser triangulator, a near-infrared camera, and a ratio pyrometer is utilized to measure the melt pool height, area, and temperature in Laser Powder DED. However, the recorded signals by camera proved to be noisy. Prediction of melt pool temperature, track geometries and hardness based on the process parameters in Laser Powder DED processes is evaluated in [20], where melt pool temperature data was obtained using

an optical fiber infrared thermometer. In [21], the utilization of a high-speed camera enabled the observation and analysis of bubble generation occurring within the melt pool during laser cladding processes. Moreover, single process parameter was utilized to monitor and control the bead or clad height, melt pool width, area, and temperature [17,22-24]. One limitation of the introduced control methods was their reliance on a single process parameter, which may restrict their effectiveness in fully capturing and regulating the complex dynamics of the process.

The limitation of relying on a single process parameter for control in DED processes has prompted researchers to explore the development of monitoring and control models that consider multiple process parameters. Artificial Intelligence (AI) methods offer the potential to incorporate multiple parameters and model DED processes. The development of Machine Learning (ML) and Convolutional Neural Networks (CNN), as subsets of AI-based methods, has had a significant impact on AM by enabling advanced data analysis, image recognition, and predictive modeling, leading to improved process monitoring, control, quality assurance, and material optimization [2]. In monitoring L-DED processes, multimodal strategies were used. In [25], different ML methods, including NNs, predicted process quality and defects. However, the accuracy and the impact of the accuracy of extracting geometric features from CCD camera-captured melt pool images were not thoroughly analyzed.

CNNs, widely applied in image and signal processing domains, effectively transform input image features into abstract representations, facilitating precise regression, classification, and segmentation tasks across diverse applications [18,26]. In the AM, CNNs have been deployed for classification [27,28], detection [29,30], segmentation [31], and regression [18,32-35] applications. For instance, in [28], a CNN-based classification algorithm was utilized to classify and to detect porosity with 91.2 % accuracy. In [30], automatic defect detection and classification in Wire Arc Additive Manufacturing processes were assessed. The proposed datasets, incorporating enhanced images, achieved a Recall criterion surpassing 88 %, indicating a high level of accuracy in correctly identifying relevant information. In [18,32,33], melt pool images were fed into a CNN and the output was fed alongside with melt pool temperature or power, voltage, and one-hot-encoded three phases (accelerating, stable, decelerating) to a fully connected Neural Network (NN) to predict the bead geometries and process parameters. In [31], it has been tried to develop a pixel-wise classification network as a segmentation algorithm to predict the welding speed and laser power. However, the accuracy and speed of the segmentation model were not well discussed since the main aim of the research was on the prediction accuracy of laser power and welding speed.

The existing literature reveals a knowledge gap regarding the comprehensive consideration and monitoring of the melt pool, its geometrical characteristics in AM processes, and the lack of correlation between these details and process inputs and outputs. Previous studies mainly relied on feeding image data without explicitly specifying the melt pool and leveraging other parameters, while the significance of melt pool characteristics remained undetermined. Furthermore, limited research has explored feature extraction and segmentation methods for melt pool monitoring in AM processes, with a lack of discussion on the accuracy, speed of the utilized architectures, and the aforementioned correlations. In this study, we develop a segmentation method based on You Only Look Once (YOLO) architecture for LW-DED processes.

The paper contributes by: (i) conducting preliminary tests using a Central Composite Design (CCD) of experiments to determine feasible process parameter ranges in a single layer LW-DED process, (ii) annotating acquired image data for YOLO-based algorithms and providing a comprehensive dataset that can be used by researchers in this field, (iii) training and testing performance of the proposed architectures in terms of speed and accuracy, (iv) defining a stability criterion for melt pool characteristics, identifying and distinguishing types of instabilities and filtering the results accordingly, and (v) conducting statistical analyses
on the sensitivity of the melt pool to process and response parameters.

The present study is structured as follows: Section 2 outlines the research methodology, including experimental and theoretical details. Section 3, discusses the performance of the proposed YOLO-based methods, stability analysis, statistical analysis, and bead geometry prediction model. Finally, Section 4 concludes with remarks on the presented solution.

## 2. Methodology

This section is divided into three parts. Section 2.1 provides the details regarding the devices, sensors, and measurement facilities by which the experiments on LW-DED are conducted. In Section 2.2, the details regarding the plan for conducting experiments are provided. Section 2.3 is assigned to describing the utilized CNN-based segmentation architecture for melt pool monitoring, the related details for its area calculation and sensitivity assessment of the acquired data from the proposed method to a variation of process parameters and mechanical characteristics of the printed samples.

### 2.1. Experimental setup

In the context of the LW-DED process, two different setups have been considered:

#### 2.1.1. In-situ measurements and manufacturing setup

The experimental setup employed for the study, as it is shown in Fig. 1(a), involves an Aerotech 3-axis gantry with a working area of 300 × 300 mm, which is outfitted with a Fraunhofer CoaxWire laser welding head and a Dinse FDE 100 L wire feeder. The energy source utilized in the experiments is a 3 kW Coherent Corelase fiber laser. It is worth mentioning that the welding head utilized in this study is a direction-independent head with coaxial wire feeder and 3 laser beams converged from three peripheral points with a difference of 120° on the focal point.

To capture the LW-DED process, a Cavitar welding camera model C300 is affixed to the setup. The camera is mounted at a default working distance of 200 ± 15 mm and has a maximum frame rate of 100 frame per second (fps). The camera's configuration, as illustrated in Fig. 1(b and c), entails its placement at a 45° angle with respect to the central axis of the welding head.

In this study, a UTP A 6222Mo-3 wire feedstock material with a diameter of 1.14 mm, identified as alloy 625 is selected, and its chemical

composition, proposed by the wire manufacturer, is presented in Table 1. 10 mm plates of 316 L (1.4404) SS according to EN10088-2 and S355 (1.0577) low carbon steel according to EN10025-2 are selected as substrates. The study aims to analyze single beads of Inconel 625 with a length of 35 mm (Fig. 2(c)), which are printed under various process parameter levels without in-situ or ex-situ thermal treatment, using the selected wire feedstock material.

#### 2.1.2. Ex-situ weld bead geometry measurements

Once all the experiments were conducted, the printed weld beads underwent laser scanning using a Romer Absolute Arm with an integrated laser scanner model RA-7525 SE (Fig. 2(a)), which boasts a 2.5 m measuring range, 0.020 mm point repeatability, and 0.029 mm volumetric accuracy. The scanner was utilized for point cloud inspection and the calculation of bead geometry measurements, including bead width, height, and area.

To conduct cross-sectional macrostructural examinations, the samples underwent a sample preparation procedure involving grinding and polishing with 1 µm diamond particles. The objective was to achieve a smooth and consistent surface on the samples. Subsequently, a 2% Nital etchant was applied to the samples to reveal the microstructure and enhance the differentiation between the weld bead and the base materials. The primary goal was to facilitate interpretation for image analysis, specifically in the identification of contours related to penetration depth and heat affected zone. The samples were then subjected to analysis using a Leica MZ7.5 microscope, and the resulting images were further analyzed using the ImageJ software [36]. Fig. 2(d) provides details regarding the calculated areas, width and height for bead, penetration and heat affected zones in printed beads.

### 2.2. Experimental plan

To conduct the experiments, three categories of parameters are relevant. (i) The first category, known as process parameters or input parameters, is established to facilitate the experiments. Table 2 lists the viable ranges and the levels of assumed Design of Experiments (DoE) for process parameters, including WFR, TS, and LP, after the completion of preliminary screening tests. (ii) Response parameters, or output parameters which are measured as the outcome of the experiments. Bead height, width, area, penetration height, and area, and the area of the heat-affected zone (HAZ) are considered as response parameters. (iii) Feedback parameters help to use in-situ measurements as the criteria for evaluating the outputs. Melt pool images and videos acquired from a

<figure><img src="image_5.png" /><figcaption>Fig. 1. Experimental setup (a) LW-DED setup, (b) schematic view of camera settings, (c) camera setup implemented on LW-DED.</figcaption></figure>
<table><caption>**Table 1**

Chemical composition of the wire.</caption><thead><tr><th colspan="11">Wire chemical composition - UTP A 6222Mo-3, wt%</th></tr><tr><th>C</th><th>Si</th><th>Mn</th><th>Ni</th><th>Cr</th><th>Mo</th><th>Cu</th><th>Ti</th><th>Al</th><th>Nb</th><th>P</th><th>S</th><th>Fe</th><th>Ta</th></tr></thead><tbody><tr><td>&lt;0.01</td><td>0.12</td><td>0.02</td><td>64.66</td><td>22.30</td><td>8.71</td><td>0.02</td><td>0.22</td><td>0.13</td><td>3.45</td><td>0.004</td><td>0.001</td><td>0.29</td><td>0.004</td></tr></tbody></table>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>**Fig. 2.** Details of (a) laser measurement device with preliminary test plates, (b) CCD DoE for LW-DED experiments, (c) the scanned CCD experiments on S355 and SS substrates, and (d) measurements in macrostructural examinations.</figcaption></figure>

<table><caption>**Table 2**

Process parameters feasible ranges and CCD DoE levels.</caption><thead><tr><th>Process parameters</th><th>Range</th><th>Levels in CCD DoE</th></tr></thead><tbody><tr><td>A: WFR (m/min)</td><td>[0.4 - 1.1]</td><td>[0.40, 0.54, 0.95, 0.74, 1.09]</td></tr><tr><td>B: TS (mm/s)</td><td>[3 - 8]</td><td>[3, 4.01, 5.5, 6.99, 8]</td></tr><tr><td>E: LP (kW)</td><td>[1.4 - 2.1]</td><td>[1.46, 1.6, 1.8, 2, 2.13]</td></tr></tbody></table>

welding vision system (Cavitar C300) at 30–70 fps and 1440 × 1080 pixels resolution are the proposed feedback data.

For each substrate material, a CCD DoE is considered. CCD is a statistical method for experimental design proposed by Box and Wilson in 1951 [37]. It uses a set of experimental conditions to fit a second-order response surface model, enabling optimization of input parameter settings and investigation of their effects on the response variable [38,39]. The incorporation of DoE in combination with Deep Neural Networks (DNNs) enables a systematic and efficient exploration of the parameter space in LW-DED experiments, ensuring comprehensive coverage and optimization of process variables while leveraging the capabilities of DNNs for accurate modeling.

For a CCD DoE with 3 process parameters and 5 center point replicates, a total of 19 data points is required, with 14 data points at the factorial and axial positions and 5 replicates at the center point to estimate pure error and assess response surface curvature. Fig. 2(b) shows the CCD DoE experimental levels for the proposed process parameters. As it is demonstrated in Fig. 2(c), the mentioned 19 single-bead experiments, with a length of 35 mm, were conducted on the aforementioned substrates using alloy 625 wire feedstock material with a diameter of 1.14 mm and varying levels of process parameters. Pure Argon was selected as shielding with gas flow of 12 l/min. To ensure the independence of the printed beads, a 10 mm distance was maintained between them. Additionally, after printing each bead, the process was paused to allow it to cool down to ambient room temperature.

## 2.3. Theoretical details

This study evaluated the correlation between process parameters and responses through the application of CNN-based methods for monitoring

melt pool behavior, and predicting bead geometries using the extracted melt pool characteristics obtained through the segmentation methods. To accomplish this objective, the theoretical underpinnings of these methods are presented in this section.

### 2.3.1. Segmentation architectures and model selection rationales

Image segmentation involves classifying pixels individually and has three subsets: semantic, instance, and panoptic segmentation. Semantic assigns object class labels to each pixel, instance identifies and segments individual objects, while panoptic combines both. For melt pool monitoring, instance segmentation is ideal, offering detailed analysis without panoptic's exhaustive details. Fully supervised instance segmentation methods can be categorized into three groups based on the stages required for object positioning and mask generation: multi-stage, two-stage, and single-stage methods. Multi-stage and two-stage methods involve sequential object positioning and mask generation, whereas single-stage methods enable simultaneous positioning and mask generation. Two-stage and multi-stage methods often prioritize accuracy over computational cost [40]. In contrast, single-stage object detection and segmentation methods achieve a more favorable balance between speed and accuracy [41].

Single-stage methods prioritize high accuracy within real-time constraints, even if it involves a slight reduction in accuracy, making them a suitable choice for melt pool monitoring. In single-stage instance segmentation methods, both prototype mask generation and per-instance mask coefficient production occur simultaneously. This approach prioritizes the equal importance of location information and mask representation, distinguishing it from two-stage and multi-stage methods where they are interdependent. The trade-off between speed and accuracy in single-stage methods may slightly reduce accuracy but prioritizes achieving the highest accuracy within low latency or real-time constraints.

Due to the scarcity of research addressing the performance of the melt pool segmentation methods, the evaluation of these models' predictive performance on benchmark datasets is being taken into account. Table 3 presents the performance results for a selection of the most renowned single and two-stage methods on the COCO dataset [42]. However, it's worth noting that two-stage methods tend to have slower
<table><caption>**Table 3**
The performance of the most well-known single and two-stage segmentation methods on COCO dataset.</caption><thead><tr><th>Model</th><th>Type</th><th>Backbone</th><th>dataset</th><th>AP</th><th>FPS</th><th>Ref</th></tr></thead><tbody><tr><td>Mask R-CNN</td><td>Two-stage</td><td>DetNet-59</td><td>COCO</td><td>AP = 37.1</td><td>5</td><td>[43]</td></tr><tr><td>FPN</td><td>Two-stage</td><td>FRCN</td><td>COCO</td><td>AP = 36.2</td><td>5</td><td>[44]</td></tr><tr><td>SSD</td><td>Single-stage</td><td>VGG-16</td><td>COCO</td><td>AP = 23.2</td><td>46</td><td>[45]</td></tr><tr><td>RetinaNet</td><td>Single-stage</td><td>ResNet-101-FPN</td><td>COCO</td><td>AP = 34.3</td><td>11</td><td>[46]</td></tr><tr><td>YOLOv5</td><td>Single-stage</td><td>Modified CSPv7</td><td>COCO</td><td>AP = 23.2</td><td>45</td><td>[47]</td></tr><tr><td>YOLOV8</td><td>Single-stage</td><td>EfficientNet, ResNet, and CSPDarknet</td><td>COCO</td><td>AP = 53.9</td><td>280</td><td>[40]</td></tr></tbody></table>

processing speeds, making them less suitable for real-time applications like melt pool monitoring. In most cases, two-stage methods exhibit higher accuracy compared to their single-stage counterparts. Nevertheless, recent versions of the YOLO model, particularly YOLOv8, have successfully struck a more favorable balance between accuracy and processing speed. The computational speed of YOLOv5 and YOLOv8 is sufficiently high to support real-time monitoring objectives while maintaining impressive accuracy levels on benchmark datasets. As a result, YOLOv5 and YOLOv8 are considered two promising versions of the YOLO framework for applications in melt pool monitoring.

### 2.3.2. Introducing you only look once (YOLO)

YOLO is a popular real-time object detection algorithm and instance segmentation based on a single convolutional neural network that simultaneously predicts bounding boxes and class probabilities for each

object in an image. The first version of YOLO was introduced in 2016 [41] and since then several improved versions have more parameters, designed to achieve higher accuracy. Fig. 3 showcases the architecture of a segmentation head, which combines the object detection capabilities of YOLOv8 with an added segmentation branch for simultaneous object detection and segmentation.

Within this study, each YOLO architecture is analyzed through the examination of two distinct variations: the smaller version (s) and the larger version (l). Comprehensive information outlining the dissimilarities between these proposed architectural versions is provided in Table 4. Evidently, the small and large versions have varying numbers of layers and parameters.

### 2.3.3. Dataset

The crucial components of a deep learning-based approach are the data preparation and training processes, particularly in image processing applications where image annotation plays a central role. Given the lack of publicly available imagery datasets in the domain of LW-DED, it is necessary to perform experiments to generate suitable data for these processes. To this end, two sets of experiments were conducted on two different substrates, namely SS and S355. The procedure for experiments, data collection and training, testing and validation is illustrated in Fig. 4. Each experiment set consisted of a CCD DoE with three process

<table><caption>**Table 4**
Number of layers and trainable parameters for the proposed architectures.</caption><thead><tr><th>YOLO version</th><th>Number of layers</th><th>Number of parameters</th></tr></thead><tbody><tr><td>YOLOv5s</td><td>225</td><td>7,408,214</td></tr><tr><td>YOLOv5l</td><td>379</td><td>47,499,542</td></tr><tr><td>YOLOv8s</td><td>261</td><td>11,790,483</td></tr><tr><td>YOLOv8l</td><td>401</td><td>45,936,819</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>**Fig. 3.** Yolo v8 architecture.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Fig. 4. Dataset collection and melt pool monitoring procedure.</figcaption></figure>

parameters and five repetitions at the center of the box, in addition to a set of 20 random tests within the specified range listed in Table 2.

All images captured during each test, from one second before deposition to one second after, were included in the dataset. Dataset includes 10 % of images captured before and after deposition, 20 % of images captured at the beginning and end of deposition (non-steady states for printing), and 70 % of captured images during the steady depositing phase. The dataset was created by selecting the same number of images from each experimental test, while ensuring that the proportions of data from each test in the dataset were maintained. After obtaining a dataset of more than 1500 images for each substrate, 25 % were reserved for validation and 75 % for training the networks. Testing data, obtained from independent random tests, comprised 300 images for each substrate. The dataset underwent annotation for malt pool boundaries using the labelme library [48], a Python package that provides an image annotation tool as well as functionality for reading, writing, and converting the resulting JSON format. The dataset is available at [49].

### 2.3.4. Monitoring system evaluation and sensitivity assessment

An instance segmentation model is a type of computer vision model that can identify and segment individual objects within an image. However, instance segmentation models can also be used as object detection models since they can identify and localize objects in an image. Such models do this by generating pixel-level masks than distinguish between different instances of same object class. By using an instance segmentation model for melt pool monitoring, precise pixel-level masks of the melt pool region can be obtained. This can provide valuable insights into the behavior and properties of the melt pool during printing processes. In this study, YOLO-based instance segmentation models are being used primarily for melt pool segmentation. The models are trained on the dataset of images obtained from LW-DED

process that contain examples of annotated melt pools, and they learn how to classify each pixel in each image. Given our study's primary emphasis on segmentation, and the exclusive consideration of a single object class i.e., melt pool, it is obvious that the performance criteria for classification and detection including classification precision and recall, exhibit remarkably high values close to 100 percent. This outcome is anticipated and, therefore, no further assessment of these criteria is warranted, aligning with the study's core focus on segmentation.

Here are the main evaluation criteria for segmentation algorithms: (i) Segmentation loss: In YOLO models, the binary cross-entropy (BCE) loss is employed as an evaluation criterion for image segmentation, as expressed in Eq. (1):

$$ \text{BCE loss} = -y\log(\hat{y}) - (1-y)\log(1-\hat{y}) \quad (1) $$

In this equation, 'y' denotes the ground-truth binary label for a pixel, signifying its association with either the object of interest (in this study, the melt pool) or the background. 'ŷ' corresponds to the predicted probability assigned to the object of interest for that specific pixel, achieved through the application of the sigmoid activation function. The sigmoid activation function transforms the model's output into a range of [0, 1] for each pixel. This is essential for the BCE loss calculation, as it facilitates the comparison between the predicted probability of the positive class (in this case, the melt pool) and the corresponding ground-truth binary label for that pixel.

The BCE loss quantifies the disparity between the predicted probabilities of the model and the true binary labels, guiding YOLO models during training to minimize this discrepancy. This optimization process contributes to the achievement of accurate segmentation of the object of interest within the image. The exact BCE loss values may depend on the dataset and model architecture, but a decreasing BCE loss indicates that the model is learning and improving its segmentation performance.

(ii) Mean Average Precision (mAP): In instance segmentation algorithms, the mAP is a widely used evaluation metric. mAP combines two fundamental metrics: Precision and Recall, both of which are calculated separately for each class (in our study, the class of interest is the melt pool). Precision (Eq. (2)) measures the accuracy of the model's predictions for a given class. It quantifies the ratio of true positive instances (correctly predicted instances of the class) to the total predicted instances (true positives and false positives).

$$ \text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}} \quad (2) $$

On the other hand, Recall assesses the model's ability to detect all actual instances of the class. It calculates the ratio of true positive instances to the total actual instances (true positives and false negatives). The formula for Recall is provided in Eq. (3)).

$$ \text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}} \quad (3) $$

mAP is a comprehensive metric that considers the precision-recall trade-off across various confidence thresholds. It provides a single summary score reflecting the overall model accuracy. To compute mAP, precision values are averaged across different recall levels, each associated with a specific Intersection over Union (IoU) threshold. In YOLO segmentation models, IoU is used to evaluate the accuracy of object segmentation. It measures the overlap between the predicted and ground truth segmentation masks, with a value of 1 indicating a perfect overlap. Typically, a detection is considered positive when the IoU exceeds a predefined threshold, often set at 0.5 or higher [50].

The standard mAP metrics of instance segmentation are defined as Eqs. (4) and (5) [51]:

$$ mAP(y, \bar{y}) = 1/N_c \sum_c A(PR - curve) \quad (4) $$
$$PR - curve = \sum_{a} \sum_{ij} (y_i, \bar{y}_j) \cdot I[IoU(y_i, \bar{y}_j) > \alpha] \quad (5)$$

Where y and $\bar{y}$ denote the predicted and ground-truth instance masks, respectively. $N_c$ represents the total number of object classes under consideration. The function A(·) calculates the area under the Precision-Recall curve, while the PR-curve refers to the smoothed precision-recall curve. I(·) denotes the indicator function. Lastly, the symbol '$\alpha$' signifies the Intersection over Union (IoU) threshold, ranging from 0.5 to 0.95 with increments of 0.05.

Following the evaluation of deep learning-based segmentation techniques, the subsequent objective is to assess the sensitivity of process and response parameters to the segmented melt pool data derived from Cavitar camera images. This sensitivity analysis is conducted utilizing statistical methods rooted in Analysis of Variance (ANOVA), encompassing the utilization of the coefficient of determination (R²) for sensitivity evaluation, as well as the examination of p-values derived from hypothesis tests and correlation matrices to elucidate relationships within the acquired melt pool data and parameters [37].

Significant sensitivity to experimental parameters is indicated by R² values exceeding 0.80, providing insights into the impact of the independent variables, represented by melt pool segmentation data, on the dependent variables, encompassing process parameters and bead geometries. Furthermore, p-values from ANOVA are employed to rigorously evaluate the statistical significance of observed effects, with a threshold of 0.05 signifying a level of statistical significance denoting a 5 % or lower probability of results arising from random chance. Additionally, correlation matrices are generated using the Pearson method, with coefficients represented by "r" values spanning from -1 to +1. These "r" values quantify both the strength and direction of linear relationships between variables, facilitating the identification of associations among them [52]. These integrated statistical techniques collectively serve to quantify parameter influences and unveil latent patterns within the dataset.

# 3. Results and discussions

This section presents the evaluation outcomes concerning the training of the proposed models, the assessment of stability and sensitivity in relation to the results obtained from the melt pool monitoring models, and the outcomes of a CNN-based bead geometry prediction model using the acquired tensor mask from the YOLO models.

## 3.1. Performance of proposed models

After the experiments based on the CCD DoE were conducted, the data collected during the experiments were evaluated. The synchronization between the acquired frames and bead geometry was performed as the camera was set to 60 fps. The proposed dataset for training and validation of the network consisted of all the images obtained from 19 tests for each substrate. The complete dataset, including the training, validation, and testing sets of images, as well as the annotation in the format of YOLO versions, has been prepared and is available at [49]. The training process for the YOLO-based methods was assumed to run for 300 epochs, but it was stopped if there were no improvements. The chosen optimizer was Stochastic Gradient Descent (SGD), coupled with a cyclic learning rate scheduler. Each cycle, representing learning rate transition from lower to upper bounds and back, was integral. The initial learning rate stood at 0.0005, with a maximum of 0.02. Training adhered to a precisely structured cyclic learning rate schedule, with 1500 evenly distributed iterations per cycle for balanced and efficient learning. Additionally, a constant of 0.985 determined the exponential range for maximum learning rate reduction.

Fig. 5 depicts the mAP and segmentation loss values obtained during the training process of both the small and large versions of YOLOv5 and YOLOv8 models applied to LW-DED experiments on SS and S355 substrates. As it can be observed, the segmentation loss for all the models has diminishing trend. The optimal epochs for training the networks on

<figure><img src="image_11.png" /><figcaption>Fig. 5. Segmentation loss and mAP50-95 values for training process of proposed (a) Yolov8-based and (b) Yolov5-based methods in LW-DED on SS and S355 substrates.</figcaption></figure>
the SS substrate were 299, 205, 299, and 218 for YOLOv5s, YOLOv5l, YOLOv8s, and YOLOv8l, respectively. Similarly, the corresponding optimal epochs for the S355 substrate were 259, 96, 169, and 174. To compare the performances of the different models, mAP is the most relevant parameter.

The overall performance of the proposed models can be assessed by referring to Fig. 5 and Table 5, which provide information on FPS and mAP50–95 values. The derived FPS value is determined by averaging the processing time, measured in milliseconds, for all frames in our experiments and subsequently converting this average into the number of frames processed per second. Generally, it is evident that all architectures exhibit superior performance when applied to the SS substrate compared to the S355 substrate. This distinction could be attributed to the surface properties and the impact of light reflection resulting from the polishing of the S355 substrate. Moreover, the YOLOv8 models demonstrate higher accuracy than the YOLOv5 models in both sets of experiments. For all the models except for YOLOv5 in segmenting melt pool in S355 substrate, the mAP50–95 criterion for large models is higher with respect to small models. However, the small models demonstrate faster processing speed for Cavitar images, as indicated by FPS. The YOLOv8l models achieve the highest accuracy, while the YOLOv8s models obtain the highest FPS among the proposed models.

To visually demonstrate the process of melt pool segmentation performed by the proposed models, Fig. 6(a) and Fig. 6(b) illustrates two sample experiments conducted on SS substrate with WFR of 0.74 m/min, TS of 5.5 mm/s, LP of 1.46 kW, and S355 substrate with WFR of 0.74 m/min, TS of 5.5 mm/s, LP of 1.8 kW respectively. The regions corresponding to the perimeters of the segmented parts are highlighted in the Figure. As anticipated, the disparities between the ground truth segmentations and the segmented parts obtained from YOLOv8 are comparatively smaller than those obtained from YOLOv5. Additionally, within the same YOLO models, the prediction accuracy is higher for the larger version of the model, resulting in segmentations that are closer to the ground truth.

## 3.2. Stability and sensitivity assessment of LW-DED experiments

Following the assessment of the performance of the proposed CNN-based models for monitoring the melt pool in LW-DED processes, it is essential to investigate the stability of the process and the sensitivity of process/response parameters to the segmented melt pool. To facilitate these assessments, image processing methods are employed to calculate metrics such as melt pool width, height, area, Center of Area (CoA), and the center of the melt pool bounding boxes. As shown in Fig. 7, for each feature (here melt pool area is considered) extracted from melt pool monitoring model, there are five different states: before deposition, starting, steady, finishing, and after deposition. Among the mentioned states, the steady state is selected for assessment as the most reliable state.

While statistical analysis primarily focuses on the steady state, it is crucial to acknowledge the occurrence of instabilities during the printing processes. These instabilities can result from various factors, including defective experiments, such as when the LP is insufficient to melt the wire and substrate effectively, leading to wire wobbling, as well as inherent uncertainties in the LW-DED processes. Therefore,

<table><caption>**Table 5**
FPS and mAP50–95 values for proposed models.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">SS substrate</th><th colspan="2">S355 substrate</th></tr><tr><th>FPS</th><th>mAP50–95</th><th>FPS</th><th>mAP50–95</th></tr></thead><tbody><tr><td>Yolov5s</td><td>73.53</td><td>0.90137</td><td>66.2</td><td>0.83222</td></tr><tr><td>Yolov5l</td><td>42.37</td><td>0.91353</td><td>42.55</td><td>0.82892</td></tr><tr><td>Yolov8s</td><td>120.48*</td><td>0.92242</td><td>114.94*</td><td>0.84686</td></tr><tr><td>Yolov8l</td><td>28.6</td><td>0.92502**</td><td>28.57</td><td>0.85357**</td></tr></tbody></table>

understanding the cause of each instability is crucial for the purpose of either filtering it out if it is inherent or implementing process control measures to prevent defects. To address this challenge, a method involving calculating the distance between the developed models' segmented melt pool CoA and the bounding box center has been employed to detect unstable melt pool conditions (Fig. 8).

For evaluating distance variations in CCD-based experiments conducted under stable conditions, Fig. 9 presents box plots for all frames of each experiment on SS substrate. These distances were calculated using the YOLOv8l model as the model with highest accuracy. It can be inferred that in a stable process, the CoA of the segmented melt pool and the center of the melt pool bounding box are consistently situated within a certain threshold in stable mode, without experiencing significant deviations compared to other stable frames. The median and mean distances for all printed beads are consistently below 100 µm, with some even below 75 µm. All printed beads exhibit upper quantiles of the distance below 100 µm, except for tests 1 and 15, where slight wire vibration during the experiments may account for deviations.

Additionally, to evaluate the statistics concerning the mentioned distance in defective tests, additional experiments were conducted. These experiments maintained a constant ED, defined as LP/TS, while proportionally adjusting the LP and TS. The WFR was kept constant throughout these experiments. Table 6 provides the details regarding the process parameter values for the experiments.

Fig. 10 provides box plots of the conducted tests based on constant ED. The results of the first four experiments closely aligned with the CCD-based experiments, all conducted under stable conditions, yielding mean and median distances below 100 µm. However, proportional reductions in LP and TS values resulted in wire wobbling defects (tests number 5 and 6). The figure clearly shows a substantial increase in mean, median, upper quantile, and maximum distances for experiments with defects, with mean and median distances exceeding 150 µm and 200 µm, respectively. Consequently, a stability threshold of 100 µm is considered for stable processes while the values more than 200 µm is a sign of defective experiments. In Fig. 7 as a sample test, the distance variations among the frames are illustrated. With the exception of transient states during the start and finish of the deposition process, the distance values consistently stay within the stable threshold. However, during these transient states, wire vibration occurs, causing the distance values to surpass the stable threshold.

After applying a 100 µm threshold to filter frames acquired during the steady state of CCD experiments, ANOVA is employed to evaluate the sensitivity of process parameters (WFR, TS, and LP) to variations in melt pool geometries (width, height, and area). Table 7 displays this data for all YOLOv8-based models, including the highly accurate YOLOv8l and the faster YOLOv8s, used with two distinct substrate materials.

The results reveal that the YOLOv8l model demonstrates higher p-values and R² values compared to the YOLOv8s model for the SS plate. Specifically, the R² values for melt pool width, height, and area are 5.23, 3.25, and 4.75 % higher, respectively, for the YOLOv8l model. A similar trend, except for melt pool height, is observed for the S355 plate. Nevertheless, the model p-values for all melt pool geometries in the process considering the stable frames are below 5 % for YOLOv8-based models. Moreover, in all cases, the processes with stable frames yields higher statistical significance when compared to normal processes that consider all frames. Additionally, for the considered substrates, the modeling of melt pool width and area proves to be more accurate and statistically significant compared to melt pool height.

For acquiring more insight into the melt pool segmentation and monitoring, the correlation between segmented melt pool geometries (including width, height, and area) and bead geometries determined by the introduced microstructural investigation system is analyzed.

Fig. 11 presents the macrographic examination of the beads' cross-sections. The solidified weld metal corresponds to the dark areas and is characterized by a mixture of Inconel 625 and fused base material. The area highlighted in blue delineate the HAZ and is characterized by the

<footnote>*The highest FPS **The best accuracy.</footnote>
<figure><img src="image_12.png" /><figcaption>Fig. 6. Visual comparison of the result of proposed networks on sample experiments conducted on (a) S355 and (b) SS plates.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 7. Different printing states shown in melt pool area calculation and the distance values between CoA and bounding boxes in a sample experiment with WFR = 0.74, TS = 5.5, and LP = 1.8.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 8. Measured distance between CoA and center of bounding box for stability assessment in a sample experiment with WFR = 0.74, TS = 5.5, and LP = 1.8.</figcaption></figure>

thermally affected part of the base material. The visual examination with optical microscope did not reveal defects such as macro-cracks, and porosities. Additionally, prior to sample preparation, surface non-destructive testing, e.g., color dye penetrant method, has been performed following the ISO 3452 standard and did not reveal any indication of lack of fusion on the surface, cracks, or open porosity.

In Fig. 12(a) the region denoted as A₃ represent the area covered by the HAZ. The determination of this parameter is specifically limited to the observations conducted on S355 base material due to its susceptibility to recrystallization, which aids in the delineation using optical microscopy. Conversely, the use of SS as substrate does not exhibit distinct identification of the HAZ, thereby justifying the exclusion of parameter A₃. In addition, the value W₀.₃ corresponds to the bead width measured at 0.3 mm from the substrate's surface. It is worth noting that the geometry of welds, such as convexity and wetting angle, variates across the different weld beads. The main advantage conferred by the inclusion of W₀.₃ in the analysis is its ability to mitigate the impact of the notable diversity in weld toe shapes. Indeed, the width of the weld bead on the surface, referred to as W, can be significantly impacted despite causing a slight change in the volume of deposited metal. Hence, a comparison between W and W₀.₃ is pertinent for the purpose of refining the measurement of weld bead width within the framework of the model. The statistics related to the correlation map in Fig. 12(b) and (c), reveal the higher correlation between W₀.₃ and melt pool geometrical characteristics compared to W.
<figure><img src="image_15.png" /><figcaption>**Fig. 9.** Box plots illustrating the computed distance between the CoA of the melt pool and the center of bounding boxes in the CCD-based experiments on SS acquired by YOLOv8l.</figcaption></figure>

<table><caption>**Table 6**
The constant ED experiment's process parameter values.</caption><thead><tr><td>Test number</td><td>WFR</td><td>TS</td><td>LP</td></tr></thead><tbody><tr><td>1</td><td>0,74</td><td>1.8</td><td>5.5</td></tr><tr><td>2</td><td>0,74</td><td>1.2</td><td>3.67</td></tr><tr><td>3</td><td>0,74</td><td>1</td><td>3.06</td></tr><tr><td>4</td><td>0,74</td><td>0.8</td><td>2.44</td></tr><tr><td>5*</td><td>0,74</td><td>0.7</td><td>2.14</td></tr><tr><td>6*</td><td>0,74</td><td>0.6</td><td>1.83</td></tr></tbody></table>

Fig. 12, presents the correlation map, based on Pearson's coefficient, between YOLOv8l-based segmented melt pool geometries (comprising melt pool height, width, and area) and bead geometrical details including bead width (W, and W₀.₃), height (h₁), area (A₁), penetration height (h₂), penetration area (A₂), HAZ area (A₃), and the overall area and height (A, and H). Fig. 12(a) presents a schematic depiction of the considered geometrical characteristics. This analysis covers experiments conducted on two substrate materials, SS (Fig. 12(b)) and S355 (Fig. 12(c)).

As it can be observed, the correlation coefficient between melt pool height and other geometrical features is not as high as the coefficients for melt pool area and width. This can also be inferred from statistics in Table 7. However, the highest related value for SS substrate is the

correlation with h₁, and for S355 substrate is the correlation with A₃ with 0.65 and 0.6 respectively. The correlation between melt pool (M. P. in the Fig. 12) area and width and geometrical features for SS substrate is more significant than the correlation for S355 substrate. As it is shown in Fig. 12(b), for SS, the correlation coefficient between M.P. area and A, A₁, h₁, and H are considered the highest in magnitude. In the mentioned substrate, A₁, A, H, h₁, and W₀.₃ are the geometrical features with correlation coefficient of more than 0.80 with M.P. area. While most of the geometrical features in SS have correlation of more than 0.7 with M.P. width, only A has a correlation value higher than 0.8. The same trend exists for S355 substrate (Fig. 12(c)), with the difference that the coefficient values are mostly between 0.5 and 0.7. The highest correlation for M. P. area in this substrate is attributed to A₃, A, H, and W₀.₃ with values of 0.71, 0.66, 0.65, and 0.62 respectively. M. P. width in S355 substrate is highly correlated with penetration features including A₂ and h₂ with values of 0.63 and 0.69 respectively.

### 3.3. Bead geometry prediction

As the final step, developing a predictive model based on the acquired melt pool characteristics by YOLO models for bead geometry as a use case of the segmentation models has been considered. The data obtained from CCD-based experiments on SS substrate have been considered. In the developed segmentation models, tensor masks are generated consisting of binary values that indicate whether a given pixel

<figure><img src="image_16.png" /><figcaption>**Fig. 10.** Box plots illustrating the computed distance between the CoA of the melt pool and the center of bounding boxes in the ED constant experiments on SS acquired by YOLOv8l.</figcaption></figure>

<footnote>* Tests with defects (wobbling wire).</footnote>
<table><caption>**Table 7**
R² and p-values of proposed YOLOv8 models for melt pool width, height, and area considering all process image frames and stable frames.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Normal process</th><th colspan="3">Process with stable frames</th></tr><tr><th>Width</th><th>Height</th><th>Area</th><th>Width</th><th>Height</th><th>Area</th></tr></thead><tbody><tr><td rowspan="3">S355 substrate</td><td rowspan="2">YOLOv8s</td><td>p-value</td><td>0.127</td><td>0.023</td><td>0.021</td><td>0.013</td><td>0.014</td><td>0.012</td></tr><tr><td>R²</td><td>0.6575</td><td>0.7783</td><td>0.7816</td><td>0.8053</td><td>0.8022*</td><td>0.8088</td></tr><tr><td>YOLOv8l</td><td>p-value</td><td>0.103</td><td>0.069</td><td>0.013</td><td>0.006</td><td>0.020</td><td>0.011</td></tr><tr><td rowspan="3">SS substrate</td><td rowspan="2">YOLOv8s</td><td>p-value</td><td>0.6765</td><td>0.7087</td><td>0.8034</td><td>0.8348*</td><td>0.7839</td><td>0.8110*</td></tr><tr><td>R²</td><td>0.174</td><td>0.137</td><td>0.046</td><td>0.007</td><td>0.025</td><td>0.012</td></tr><tr><td>YOLOv8l</td><td>p-value</td><td>0.6260</td><td>0.6504</td><td>0.7363</td><td>0.8310</td><td>0.7725</td><td>0.8095</td></tr><tr><td></td><td></td><td>R²</td><td>0.033</td><td>0.046</td><td>0.053</td><td>0.001</td><td>0.013</td><td>0.003</td></tr></tbody></table>

<figure><img src="image_17.png" /><figcaption>**Fig. 11.** Cross-sectional images using optical microscopy for experiments on S355 substrate.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>**Fig. 12.** The correlation map between segmented melt pool geometrical features and bead geometrical features acquired by macrostructural examination method for (a) schematic view of geometrical characteristics (b) SS substrates and (c) S355.</figcaption></figure>

in a frame belongs to the melt pool or not. In the predictive model, such tensors are fed to the developed CNN-based model to predict the bead geometrical characteristics, namely its width, height, and area. For the prediction model, the mask tensors acquired from YOLOv8l model as the most precise model have been considered.

The architecture of the CNN-based model is presented in Fig. 13. The

model consists of two convolutional layers, each one integrated with Rectified Linear Unit (ReLU) activation function and batch normalization. After each convolutional layer, a max-pooling layer is applied, and the model is completed with three fully connected layers. 2000 images extracted from the stable states in the conducted experiments, including CCD-based and randomly generated experiments. The training process employed the Adam optimizer as the model optimizer and Mean Square Error (MSE) as the chosen loss function. Throughout the network training procedure, a batch size of 16 instances of data was used in each iteration to implement model training and optimization.

Fig. 14 provides the details regarding the accuracy of the bead geometries predictions. Fig. 14(a) illustrates a representative prediction stemming from CCD-based experiments, featuring process parameters: WFR = 0.54 m/min, TS = 4 mm/s, and LP = 2 kW. Notably, the parameter displaying the highest level of accuracy is the bead area, as corroborated by its depiction in the correlation map found in Fig. 12.

Furthermore, Fig. 14(b) displays a box plot that provides an overview of the average MSE values across the printing of each bead for all CCD-based experiments. The computed mean values for bead area, height, and width are 0.082, 0.111, and 0.334, respectively. These findings emphasize the model's accuracy in predicting bead area, although the precision in predicting bead height is relatively lower when compared to predictions for bead width and area. In conclusion, the methodology of modeling bead geometries utilizing melt pool images has demonstrated its feasibility and effectiveness.

## 4. Conclusion

In conclusion, this study successfully conducted the detection, annotation, and segmentation of the melt pool in controlled CCD experiments within the valid range of process parameters for the LW-DED process. The utilization of YOLO-based algorithms proved effective in achieving these objectives. The obtained results underwent rigorous statistical analysis, allowing for meaningful insights. Moreover, the tensor masks generated by the segmentation models have demonstrated their effectiveness when employed in bead geometry prediction models.

<footnote>* Most accurate model.</footnote>
<figure><img src="image_19.png" /><figcaption>Fig. 13. Architecture of the developed CNN-based model for bead geometry prediction.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Fig. 14. CNN-based model prediction performance: (a) Predicting bead geometries with sample process parameters (WFR = 0.54 m/min, TS = 4 mm/s, and LP = 2 kW) on SS. (b) Box plots of average MSE for CCD-based experiments.</figcaption></figure>

The main conclusions drawn from this research are as follows:

* The integration of data from CCD DoE experiments, including in-situ and ex-situ measurements, into standardized datasets proved to be a valuable contribution. These datasets consisted of meticulously annotated images from 19 experiments on each substrate, following YOLOv5 and YOLOv8 standards. With limited comparable datasets available, the publication of this dataset provides researchers with a valuable resource, promoting further advancements in the field and serving as a robust reference for future research.

* The study examined four YOLO-based algorithms, including both large models (YOLOv5l and YOLOv8l) and small models (YOLOv5s and YOLOv8s), for training datasets. The analysis demonstrated that modeling melt pool geometries for the SS substrate performed better across the same models due to distinctive visual and surface characteristics. The most accurate melt pool segmentation models were based on YOLOv8l, achieving mAP50-95 scores of 95.5 % and 85.3 % for the SS and S355 substrates, respectively. Additionally, the smaller YOLO versions exhibited higher FPS rates, with YOLOv8s being 63.4 % and 73.4 % faster than YOLOv5s for the SS and LCS substrates, respectively.
* To ensure robust statistical analysis, it is imperative to focus on the stable deposition conditions. To achieve this, two key aspects were investigated. Firstly, the analysis concentrated on the stable phase of the deposition process, which occurs after the initial transient phase and continues until the process conclusion. This approach allowed for an examination of consistent process behavior. Secondly, an instability indicator was introduced to quantify deviations, encompassing inherent instabilities and defective tests. This was achieved by calculating the distance between the center of the melt pool bounding box and the center of the segmented melt pool area. Instances where this distance exceeded 100 µm were flagged as indicative of inherent instabilities in stable process conditions where defects do not exist. Frames exhibiting such instabilities were excluded from the statistical analysis, enhancing the accuracy of the statistical analysis of the melt pool characteristics concerning process attributes.

* The analysis of the trained YOLOv8-based melt pool segmentation models yielded important findings. Firstly, the analysis of variance confirmed higher R² values and p-values for stabilized process data compared to data from processes without stable frames. Furthermore, modeling melt pool width and area based on process parameters demonstrated superior accuracy compared to height modeling, although the height modeling results were still acceptable. For the SS substrate, YOLOv8l models showed higher significance across all melt pool geometrical features than YOLOv8s models, with R² values for large models being over 3 % higher. Similar trends were observed for the S355 substrate, except for melt pool height. Notably, YOLOv8s exhibited a p-value of 0.006 less than YOLOv8l for melt pool height, indicating a more accurate model in that specific case.

* The correlation between macrographic examination results of bead cross-sections and melt pool geometries, calculated using melt pool segmentation models, was analyzed. In the SS substrate, there was no significant correlation between height and geometrical features. However, the melt pool area showed a strong correlation with bead area and height. Furthermore, melt pool width exhibited a high correlation with bead area, width, and overall height. For the S355 substrate, the correlation values were lower compared to those for SS. Melt pool area correlated with the area of the heat-affected zone, as well as bead width, height, and area. Additionally, melt pool width showed its highest correlation with penetration area and height.

* The assessment of predicting bead geometrical characteristics using CNN-based models fed with tensor masks obtained from the segmentation models was conducted. The analysis confirmed the capability of melt pool segmentation outputs to predict bead geometries with average mean squared error values of 0.082, 0.111, and 0.334 for bead area, height, and width, respectively.

This research was limited to single beads as the precursor to apply the same DNN method in multilayer DED. Further research is planned to focus on utilizing the proposed approach as a foundation for developing model predictive controllers in multilayer additive manufacturing particularly in the context of directed energy deposition processes. Additionally, we plan to explore the integration of the proposed framework into sensor fusion platforms, enabling comprehensive and real-time monitoring systems that combine imagery, acoustic emission, and temperature sensing for numeric and categorical data using AI-based and statistical methods. These advancements would contribute to the development of more robust and efficient monitoring and control systems for enhanced process performance and product quality in AM.

## APPENDIX A. Supplementary data

The following is the supplementary material related to this article.

## CRediT authorship contribution statement

**Reza Asadi:** Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. **Antoine Queguineur:** Conceptualization, Resources, Writing – original draft, Writing – review & editing, Visualization. **Olli Wiikinkoski:** Conceptualization, Investigation. **Hossein Mokhtarian:** Formal analysis, Writing – review & editing, Supervision. **Tommi Aihkisalo:** Formal analysis, Writing – review & editing. **Alejandro Revuelta:** Formal analysis, Writing – review & editing. **Ínigo Flores Ituarte:** Conceptualization, Methodology, Investigation, Formal analysis, Writing – review & editing, Visualization, Supervision, Funding acquisition.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

Data will be made available on request.

## Acknowledgement

This research has been supported by the project TANDEM (4056/31/2021) Business Finland under the SMART EUREKA cluster on advance manufacturing program and by the project Multi-disciplinary Digital Design and Manufacturing, D2M (346874) Academy of Finland/Academy Research Fellow. Research has been conducted at the Digital Design and Manufacturing (D2M) laboratory at Tampere University in Finland with the support laboratory of Aapo Ylä-Autio and Jorma Vihinen. We would like to express our heartfelt gratitude to Erkki Lassila from Cavitar Ltd for their invaluable assistance in facilitating the execution and maintenance of the hardware (C300 camera) utilized in our research. Additionally, we extend our deep appreciation to Dragos Stan for his invaluable guidance and expertise in AI-based methods.

## Supplementary materials

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.rcim.2023.102710.

## References

[1] R. Ponche, O. Kerbrat, P. Mognol, J.-Y. Hascoet, A novel methodology of design for additive manufacturing applied to additive laser manufacturing process, Robot. Comput.-Integr. Manuf. 30 (4) (Aug. 2014) 389-398, https://doi.org/10.1016/j.rcim.2013.12.001.

[2] F. He, et al., Research and application of artificial intelligence techniques for wire arc additive manufacturing: a state-of-the-art review, Robot. Comput.-Integr. Manuf. 82 (Aug. 2023), 102525, https://doi.org/10.1016/j.rcim.2023.102525.

[3] J. Foshammer, P.V. Soberg, P. Helo, I.F. Ituarte, Identification of aftermarket and legacy parts suitable for additive manufacturing: a knowledge management-based approach, Int. J. Prod. Econ. 253 (Nov. 2022), 108573, https://doi.org/10.1016/j.ijpe.2022.108573.

[4] I. Flores, N. Kretzschmar, A.H. Azman, S. Chekurov, D.B. Pedersen, A. Chaudhuri, Implications of lattice structures on economics and productivity of metal powder bed fusion, Addit. Manuf. 31 (Jan. 2020), 100947, https://doi.org/10.1016/j.addma.2019.100947.

[5] M. Chalvin, S. Campocasso, V. Hugel, T. Baizeau, Layer-by-layer generation of optimized joint trajectory for multi-axis robotized additive manufacturing of parts of revolution, Robot. Comput.-Integr. Manuf. 65 (Oct. 2020), 101960, https://doi.org/10.1016/j.rcim.2020.101960.

[6] M. Froend, V. Ventzke, F. Dorn, N. Kashaev, B. Klusemann, J. Enz, Microstructure by design: an approach of grain refinement and isotropy improvement in multi-layer wire-based laser metal deposition, Mater. Sci. Eng. A 772 (Jan. 2020), 138635, https://doi.org/10.1016/j.msea.2019.138635.
[7] A. Queguineur, et al., Wire arc additive manufacturing of thin and thick walls made of duplex stainless steel, Int. J. Adv. Manuf. Technol. (May 2023), https://doi.org/10.1007/s00170-023-11560-5.

[8] M. Ostolaza, J.I. Arrizubiera, A. Queguineur, K. Valtonen, A. Lamikiz, I. Flores Ituarte, Influence of process parameters on the particle-matrix interaction of WC-Co metal matrix composites produced by laser-directed energy deposition, Mater. Des. 223 (Nov. 2022), 111172, https://doi.org/10.1016/j.matdes.2022.111172.

[9] M. Akbari, R. Kovacevic, An investigation on mechanical and microstructural properties of 316LSi parts fabricated by a robotized laser/wire direct metal deposition system, Addit. Manuf. 23 (Oct. 2018) 487-497, https://doi.org/10.1016/j.addma.2018.08.031.

[10] L. Sun, et al., Investigation on the process window with liner energy density for single-layer parts fabricated by wire and arc additive manufacturing, J. Manuf. Process. 56 (Aug. 2020) 898-907, https://doi.org/10.1016/j.jmapro.2020.05.054.

[11] N. Shamsaei, A. Yadollahi, L. Bian, S.M. Thompson, An overview of Direct Laser Deposition for additive manufacturing; Part II: mechanical behavior, process parameter optimization and control, Addit. Manuf. 8 (Oct. 2015) 12-35, https://doi.org/10.1016/j.addma.2015.07.002.

[12] M. Bambach, I. Sizova, F. Kies, C. Haase, Directed energy deposition of Inconel 718 powder, cold and hot wire using a six-beam direct diode laser set-up, Addit. Manuf. 47 (Nov. 2021), 102269, https://doi.org/10.1016/j.addma.2021.102269.

[13] Z. Li, et al., High deposition rate powder- and wire-fed laser directed energy deposition of metallic materials: a review, Int. J. Mach. Tools Manuf. 181 (Oct. 2022), 103942, https://doi.org/10.1016/j.jjmactools.2022.103942.

[14] M. Motta, A.G. Demir, B. Previtali, High-speed imaging and process characterization of coaxial laser metal wire deposition, Addit. Manuf. 22 (Aug. 2018) 497-507, https://doi.org/10.1016/j.addma.2018.05.043.

[15] W. Li, et al., Research and prospect of on-line monitoring technology for laser additive manufacturing, Int. J. Adv. Manuf. Technol. 125 (1-2) (Mar. 2023) 25-46, https://doi.org/10.1007/s00170-022-10758-3.

[16] M. Akbari, Y. Ding, R. Kovacevic, Process Development for a Robotized Laser Wire Additive Manufacturing. Volume 2: Additive Manufacturing; Materials, American Society of Mechanical Engineers, Los Angeles, California, USA, Jun. 2017, https://doi.org/10.1115/MSEC2017-2951. V002T01A015.

[17] L. Song, V. Bagavath-Singh, B. Dutta, J. Mazumder, Control of melt pool temperature and deposition height during direct metal deposition process, Int. J. Adv. Manuf. Technol. 58 (1-4) (Jan. 2012) 247-256, https://doi.org/10.1007/s00170-011-3395-2.

[18] N. Jamnikar, S. Liu, C. Brice, X. Zhang, Comprehensive molten pool condition-process relations modeling using CNN for wire-feed laser additive manufacturing, J. Manuf. Process. 98 (Jul. 2023) 42-53, https://doi.org/10.1016/j.jmapro.2023.05.004.

[19] S. Maffia, V. Furlan, B. Previtali, Coaxial and synchronous monitoring of molten pool height, area, and temperature in laser metal deposition, Opt. Laser Technol. 163 (Aug. 2023), 109395, https://doi.org/10.1016/j.optlastec.2023.109395.

[20] J. Gao, C. Wang, Y. Hao, X. Wang, K. Zhao, X. Ding, Prediction of molten pool temperature and processing quality in laser metal deposition based on back propagation neural network algorithm, Opt. Laser Technol. 155 (Nov. 2022), 108363, https://doi.org/10.1016/j.optlastec.2022.108363.

[21] K. Tanaka, T. Yamaguchi, Direct observation of bubble generation processes inside a molten pool during laser cladding, Surf. Coat. Technol. 447 (Oct. 2022), 128831, https://doi.org/10.1016/j.surfcoat.2022.128831.

[22] A. Fathi, A. Khajepour, E. Toyserkani, M. Durali, Clad height control in laser solid freeform fabrication using a feedforward PID controller, Int. J. Adv. Manuf. Technol. 35 (3-4) (Dec. 2007) 280-292, https://doi.org/10.1007/s00170-006-0721-1.

[23] D. Hu, R. Kovacevic, Sensing, modeling and control for laser-based additive manufacturing, Int. J. Mach. Tools Manuf. 43 (1) (Jan. 2003) 51-60, https://doi.org/10.1016/S0890-6955(02)00163-3.

[24] J.T. Hofman, B. Pathiraj, J. Van Dijk, D.F. De Lange, J. Meijer, A camera based feedback control strategy for the laser cladding process, J. Mater. Process. Technol. 212 (11) (Nov. 2012) 2455-2462, https://doi.org/10.1016/j.jmatprotec.2012.06.027.

[25] L. Chen, et al., Multisensor fusion-based digital twin for localized quality prediction in robotic laser-directed energy deposition, Robot. Comput.-Integr. Manuf. 84 (Dec. 2023), 102581, https://doi.org/10.1016/j.rcim.2023.102581.

[26] A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM 60 (6) (May 2017) 84-90, https://doi.org/10.1145/3065386.

[27] X. Chen, et al., An in-process multi-feature data fusion nondestructive testing approach for wire arc additive manufacturing, Rapid Prototyp. J. 28 (3) (Mar. 2022) 573-584, https://doi.org/10.1108/RJP-02-2021-0034.

[28] B. Zhang, S. Liu, Y.C. Shin, In-Process monitoring of porosity during laser additive manufacturing process, Addit. Manuf. 28 (Aug. 2019) 497-505, https://doi.org/10.1016/j.addma.2019.05.030.

[29] H.-W. Cho, S.-J. Shin, G.-J. Seo, D.B. Kim, D.-H. Lee, Real-time anomaly detection using convolutional neural network in wire arc additive manufacturing: molybdenum material, J. Mater. Process. Technol. 302 (Apr. 2022), 117495, https://doi.org/10.1016/j.jmatprotec.2022.117495.

[30] X. He, T. Wang, K. Wu, H. Liu, Automatic defects detection and classification of low carbon steel WAAM products using improved remanence/magneto-optical imaging

and cost-sensitive convolutional neural network, Measurement 173 (Mar. 2021), 108633, https://doi.org/10.1016/j.measurement.2020.108633.

[31] C. Knaak, G. Kolter, F. Schulze, M. Kröger, and P. Abels, “Deep learning-based semantic segmentation for in-process monitoring in laser welding applications,” in *Applications of Machine Learning*, M. E. Zelinski, T. M. Taha, J. Howe, A. A. Awwal, and K. M. Iftekhарuddin, Eds., San Diego, United States: SPIE, Sep. 2019, p. 2. doi: 10.1117/12.2529160.

[32] M. Perani, S. Baraldo, M. Decker, A. Vandone, A. Valente, B. Paoli, Track geometry prediction for laser metal deposition based on on-line artificial vision and deep neural networks, Robot. Comput.-Integr. Manuf. 79 (Feb. 2023), 102445, https://doi.org/10.1016/j.rcim.2022.102445.

[33] N.D. Jamnikar, S. Liu, C. Brice, X. Zhang, In-process comprehensive prediction of bead geometry for laser wire-feed DED system using molten pool sensing data and multi-modality CNN, Int. J. Adv. Manuf. Technol. 121 (1-2) (Jul. 2022) 903-917, https://doi.org/10.1007/s00170-022-09248-3.

[34] Q. Wang, W. Jiao, P. Wang, Y. Zhang, A tutorial on deep learning-based data analytics in manufacturing through a welding case study, J. Manuf. Process. 63 (Mar. 2021) 2-13, https://doi.org/10.1016/j.jmapro.2020.04.044.

[35] C. Gonzalez-Val, A. Pallas, V. Panadeiro, A. Rodriguez, A convolutional approach to quality monitoring for laser manufacturing, J. Intell. Manuf. 31 (3) (Mar. 2020) 789-795, https://doi.org/10.1007/s10845-019-01495-8.

[36] C.A. Schneider, W.S. Rasband, K.W. Eliceiri, NIH Image to ImageJ: 25 years of image analysis, Nat. Methods 9 (7) (Jul. 2012) 671-675, https://doi.org/10.1038/nmeth.2089.

[37] G.E.P. Box, K.B. Wilson, On the experimental attainment of optimum conditions, J. R. Stat. Soc. Ser. B Methodol. 13 (1) (Jan. 1951) 1-38, https://doi.org/10.1111/j.2517-6161.1951.tb00067.x.

[38] I.F. Ituarte, F. Nilsén, V.K. Nadimpalli, M. Salmi, J. Lehtonen, S.-P. Hannula, Towards the additive manufacturing of Ni-Mn-Ga complex devices with magnetic field induced strain, Addit. Manuf. 49 (Jan. 2022), 102485, https://doi.org/10.1016/j.addma.2021.102485.

[39] I. Flores Ituarte, S. Panicker, H.P.N. Nagarajan, E. Coatanea, D.W. Rosen, Optimisation-driven design to explore and exploit the process-structure-property-performance linkages in digital manufacturing, J. Intell. Manuf. 34 (1) (Jan. 2023) 219-241, https://doi.org/10.1007/s10845-022-02010-2.

[40] J. Kaur, W. Singh, A systematic review of object detection from images using deep learning, Multimed. Tools Appl. (Jun. 2023), https://doi.org/10.1007/s11042-023-15981-y.

[41] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: unified, real-time object detection, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, IEEE, Jun. 2016, pp. 779-788, https://doi.org/10.1109/CVPR.2016.91.

[42] T.-Y. Lin et al., “Microsoft COCO: common Objects in Context,” in Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds., in Lecture Notes in Computer Science, vol. 8693. Cham: Springer International Publishing, 2014, pp. 740-755. doi: 10.1007/978-3-319-10602-1-48.

[43] K. He, G. Gkioxari, P. Dollar, R. Girshick, Mask R-CNN, in: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, IEEE, Oct. 2017, pp. 2980-2988, https://doi.org/10.1109/ICCV.2017.322.

[44] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” 2016, doi: 10.48550/ARXIV.1612.03144.

[45] in Computer Vision – ECCV 2016 W. Liu, et al., SSD: single shot MultiBox detector, in: B. Leibe, J. Matas, N. Sebe, M. Welling (Eds.), Lecture Notes in Computer Science, vol. 9905, Springer International Publishing, Cham, 2016, pp. 21-37, https://doi.org/10.1007/978-3-319-46448-0-2.

[46] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollar, Focal loss for dense object detection, in: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, Venice, Oct. 2017, pp. 2999-3007, https://doi.org/10.1109/ICCV.2017.324.

[47] A. Kuznetsova, T. Maleva, and V. Soloviev, “Detecting apples in orchards using YOLOv3 and YOLOv5 in general and close-up images,” in Advances in Neural Networks – ISNN 2020, M. Han, S. Qin, and N. Zhang, Eds., in Lecture Notes in Computer Science, vol. 12557. Cham: Springer International Publishing, 2020, pp. 233-243. doi: 10.1007/978-3-303-064221-1_20.

[48] K. Wada, “Labelme: image Polygonal Annotation with Python.” doi: 10.5281/zenodo.5711226.

[49] R. Asadi, CCD-based melt pool annotation inLW-DED, Zenodo (2023), https://doi.org/10.5281/ZENODO.8058411. Jun. 20.

[50] M.A. Rahman and Y. Wang, “Optimizing intersection-over-union in deep neural networks for image segmentation,” in Advances in Visual Computing, G. Bebis, R. Boyle, B. Parvin, D. Koracin, F. Porikli, S. Skaff, A. Entezari, J. Min, D. Iwai, A. Sadagic, C. Scheidegger, and T. Isenberg, Eds., in Lecture Notes in Computer Science, vol. 10072. Cham: Springer International Publishing, 2016, pp. 234-244. doi: 10.1007/978-3-319-50835-1_22.

[51] W. Gu, S. Bai, L. Kong, A review on 2D instance segmentation based on deep neural networks, Image Vis. Comput. 120 (Apr. 2022), 104401, https://doi.org/10.1016/j.imavis.2022.104401.

[52] J.I. Piovani, The historical construction of correlation as a conceptual and operative instrument for empirical research, Qual. Quant. 42 (6) (Dec. 2008) 757-777, https://doi.org/10.1007/s11135-006-9066-y.