# Multi-Source Image Fusion Based Regional Classification Method for Apple Diseases and Pests
## 2 Materials and Methods
### Multi-Label Classification Model for Apple Disease and Pest Areas
#### 2.3.2 Data Fusion Layer  
In the canopy images of apple diseases and pests, RGB images can reflect the texture information of vegetation, while vegetation indices reflect the physiological characteristics of vegetation. To make more comprehensive use of this information and fully excavate the common features beneficial to the prediction of apple diseases and pests, the above two kinds of images need to be fused. In the multi-source image fusion task, according to different stages of fusion, the fusion method can be divided into data-level fusion, feature-level fusion, and decision-level fusion, which, respectively, correspond to the original data source stage, feature representation stage, and decision-making stage. The RGB and multispectral images of the canopy apple pests and diseases were spatially aligned due to simultaneous shooting at the same time and position, thus ensuring the spatial consistency between the RGB image and the calculated vegetation indices. Data-level fusion can ensure data integrity and help reduce the situations in which details are ignored due to the macro features obtained from feature extraction. This approach can mine the common features beneficial for disease and pest prediction from both types of data more effectively and can comprehensively describe the growth status and disease and pest conditions of apple trees. Therefore, the data-level fusion method was adopted to combine the two kinds of image data. The process can be expressed as Equation (6). \[X=Cat(X_{RGB},Dot(X_{VIs},W_{RF-ML})) \tag{6}\]  
Figure 5: The overall structure of AMMFNet. wherein \(X\) is the fused data, \(X_{RGB}\) is the RGB images, \(X_{VIs}\) is the vegetation indices, and \(W_{RF-ML}\) is the importance weight calculated by RF-ML. \(Cat(\cdot)\) represents the data tensor stitching in the channel dimension, and \(Dot(\cdot)\) represents the data tensor and weight vector dot product in the channel dimension. The fused data need to further integrate the texture information contained in the RGB image and the physiological information reflected by the vegetation index. Each channel of the fused data may carry features related to diseases and insect pests. For example, the red channel of the RGB image highlights the edge and veins of the leaves, and the green channel can better capture the changes in the surface texture and details of the leaves. The vegetation index can calculate the correlation with diseases and pests through the feature selection algorithm as shown in Figure 6. MTVI2, SR, and NLI can capture the changes in apple leaves caused by the invasion of diseases and insect pests such as brown spot or gray spot by monitoring physiological parameters such as vegetation growth status, chlorophyll content, and coverage, which has strong sensitivity to diseases and insect pests. Therefore, to reintegrate the pest and disease-sensitive channels in the data before inputting into the model, channel attention was used to ensure that the model automatically focuses on the feature channels sensitive to pests and diseases, fully explores the complementary information among multi-source features, and enhances the final feature representation. Channel attention [59] is a common attention mechanism in deep learning, aiming at improving the model attention to different channels of the input data. By learning the weights of each channel, the model can adaptively adjust the importance of the channels to capture the features of the input data more effectively. Specifically, the channel attention calculates the global average-pooling and global max-pooling of the input image, then passes through a shared multi-layer perceptron (MLP), and finally adds the two output results and calculates the channel attention score through the activate function. It then multiplies the attention score with the original image tensor point to enhance the results. The calculation is shown in Equation (7).