# Video Interpolation-Based Multi-Source Data Fusion Method for Laser Processing Melt Pool
## 2 Experiments and Data Preprocessing
### Data Preprocessing  
In Section 2.1, the collected multi-source data of the melt pool had three issues. Firstly, the data contained a significant amount of noise, such as the temperature distribution of the cladding head included in the infrared temperature distribution data and the noise generated by light reflection in the visible-light data. Secondly, the multi-source data had not been registered. Due to the different installation positions of the infrared thermal camera and the high-speed camera, it is difficult to precisely align features from different sources during data fusion; hence, data registration is a necessary step. Lastly, deep learning models require a substantial amount of data support, and the volume of data obtained was insufficient to meet the needs of network model training. In response to these issues, this study proposes a multi-source data pre-processing method, as shown in Figure 2. This method includes data cleaning, data registration, ROI extraction, and data augmentation. 1. Data Cleaning: In this study, the data cleaning process consisted of two main stages. For visible-light data, the specular reflection part was removed using a manually designed IR mask. The design of the IR mask is shown in Figure 2, marking the cladding nozzle area as True and the rest as False. For infrared temperature distribution data, a threshold segmentation algorithm was first applied, with a threshold set to 90, to retain the temperature data of the melt pool area and droplet splash. Then, an 8-connected region analysis method was used to ensure that the melt pool area and the cladding head area were not connected in the image and were both complete. Finally, the temperature distribution of the cladding head was removed, and the temperature distribution of the melt pool was retained. 2. Data Registration: This study employed affine transformation to process images obtained from the infrared thermal camera to achieve registration between the visible-light and the infrared thermal camera processing areas. The affine transformation formula is as follows: \[ \begin{bmatrix}x^{\prime}\\ y^{\prime}\\ 1\end{bmatrix} = \begin{bmatrix}R_{00}&R_{01}&R_{02}\\ R_{10}&R_{11}&R_{12}\\ 0&0&1\end{bmatrix} \cdot \begin{bmatrix}x\\ y\\ 1\end{bmatrix} \] (1)where \(\left[x\quad y\quad 1\right]\) represents the coordinates of a certain pixel in the original infrared image, \(\left[x^{\prime}\quad y^{\prime}\quad 1\right]\) represents the corresponding coordinates after affine transformation, and \(R_{ij}\) denotes the parameters in the affine transformation matrix. 3. ROI Extraction: After the affine transformation was completed, we proceeded with ROI extraction. Since the center of the transformed infrared image still corresponds to the cladding area, to ensure that the ROI regions of the visible-light and infrared images correspond to the same cladding area, the position of the laser light source is taken as the center of the ROI region. Based on the temperature distribution of the melt pool area, the resolution of the infrared image ROI region is set to 200 \(\times\) 200 to maintain the characteristics of the temperature distribution of the melt pool and its surroundings, reducing feature loss. Considering the difference in spatial resolution between the high-speed camera and the infrared thermal camera, to ensure consistency in the spatial resolution of the multi-source data during image fusion and to retain as much image information as possible, the infrared data are upsampled to align spatially with the visible-light video data. The resolution of the visible-light image ROI region is also set to 200 \(\times\) 200. To address potential data distortion during the upsampling process, this study included distortion compensation in the image fusion module to ensure the authenticity of the fused images. 4. Data Augmentation: The number of infrared data collected in the experiment was relatively small, only 347 images, while the number of visible-light data was larger, approximately 694 images.