## 3 Methodology
### Multimodal Fusion Model
#### 3.2.2 Video-PPG Fusion Model  
In cross-modal data feature fusion, straightforward fusion methods such as feature stacking or concatenation are commonly applied directly to classification tasks. However, standard attention fusion methods, which are based on global information to compute fusion weights, may not be effective for fine-grained small features in different modes of data. This can easily lead to inconsistencies in data scale (both spatial and temporal) and semantics between different modes of data, consequently degrading the performance of  
Figure 9: The architecture of the cascade attention-based facial expression recognition network model. We note that X represents the input to the network; Conv-1, Pool-1, Conv-2, Conv-3, Conv-4, and Conv-5 are the inner layers of the ResNeXt network; Pyramid denotes the pyramid feature extractor; Fpta(x) denotes the output of the pyramid feature extractor; Fconv-4(x\({}^{\text{T}}\)) and Fconv-5(x\({}^{\text{T}}\)) represent the output characteristics of Conv-4 and Conv-5 of the ResNeXt network, respectively; Fpce(x\({}^{\text{T}}\)) denotes the input characteristics of the cascaded attention module; and \(\oplus\) denotes the superimposed fusion operation of the features. The face image in this figure is from the CK+ database “S113”. network models. To address this, a hierarchical refinement of the entire prediction process is sought. Firstly, the video and PPG modules independently focus on their respective tasks to learn superior data representations. Secondly, a model-level fusion strategy is adopted, which introduces feature concatenation or stacking during the feature processing stage to facilitate feature fusion for subsequent tasks [36; 37]. To better integrate semantic and scale-inconsistent features, this paper introduces the multi-scale channel attention method proposed by [PERSON] [11]. This network uses the channel attention mechanism to perform multi-scale feature fusion at various levels of the backbone network and is applicable to feature fusion for single-modal signals [11]. In this study, their Attention Fusion Network (AFF) has been adjusted and optimized, and a parallel connected attention aggregation network (PCAN) has been constructed for learning emotion recognition. Specifically, PCAN utilizes a scale attention module to fuse features at different scales locally and globally, employing two branches of different scales to extract channel attention weights and emphasizing contextual information aggregation at scales, ensuring that the data scales and semantics of different modalities remain consistent. The parallel connection input of multimodal features endows the network with comprehensive perceptual capabilities for various granularity features in different modal data. In Figure 10, \"Composition\" refers to an additive operation, while \"Multiply\" indicates a multiplicative operation. The video signal \(F_{y}\) and PPG signal \(F_{p}\) are input into PCAN for feature aggregation. PCAN primarily utilizes the Multi-Scale Attention Module (MS-CAM) to process the two types of modal data separately, as shown in Figure 10. This module employs two branches of varying scales to extract channel attention weights. One branch employs GlobalAvgPooling to capture global attention weights, while the other directly uses Pointwise operations to extract local channel attention weights. These two sets of weights effectively distinguish the importance of the original single-modal data features. The specific calculation process of the multi-scale attention modules is detailed in Equations (12) and (13). \(L(X)\) represents the first branch in Figure 10, and \(g(X)\) represents the second branch. \[L(X)=\beta(PWConv_{2}(\delta(\beta(PWConv_{1}(X))))) \tag{12}\]  
\[X^{{}^{\prime}}=X*\sigma(L(X)+g(X)) \tag{13}\]  
PCAN is built on multi-scale attention modules and employs multi-branch parallel connections to aggregate features from different modalities. The network aggregation method is presented in Figure 11.  
Figure 10: Multi-scale attention module diagram.