## 1 Introduction  
Emotions play a crucial role in human perception, attention, control, and decision making [1]. The application of affective computing technology in the field of education has led to the emergence of affective e-learning, which leverages emotional data to enhance learning effectiveness [1, 2]. Human expression of emotions is often accompanied by fluctuations in physiological or behavioral characteristics. Researchers explore these emotions through various modalities of data (text, speech, video, physiological signals, etc.). However, not all signals can be applied to online learning environments. By comparison, utilizing cameras and wearable devices can help achieve good long-term monitoring [3]. Among them, EEG (electroencephalogram) signals and ECG (electrocardiogram) signals have good objectivity, but the acquisition sensors are expensive, the operation is complex, and there is also significant interference to learners; photoplethysmography (PPG) enables straightforward and economical data collection without compromising student comfort. Consequently, the integration of video signals and PPG signals significantly mitigates the impact of external factors on learning emotion recognition, facilitating long-term assessments of learners [4]. However, facial emotion recognition in video sequences is still susceptible to objective factors such as lighting, blurring, occlusion, and others that affect image quality [5; 6], while wearable devices for PPG signal collection can potentially compensate for the limitations of facial emotion recognition [7]. Nonetheless, despite PPG's non-invasive nature and its ease of collection, it is vulnerable to factors such as complex human movements, ambient lighting, device-skin contact pressure, and subject skin color, necessitating improvements in emotion recognition accuracy [8]. Relevant research on PPG signals can provide valuable insights; for example, [PERSON] and others used wavelet transform to process PPG signal data, so as to suppress the noise in the PPG data and improve the quality of the PPG data [9]. In our previous work [10], we realized the attempt to extract a heartbeat signal from rPPG based on wavelet transform spectrograms. It was also proved that the wavelet multi-scale signal is helpful for enhancing feature information and in training classification models. In addition, the multi-sensor data fusion technology also provided inspiration to the following learners' emotion recognition studies [11; 12]. Building on these considerations, this article aims to refine the PPG-based emotion recognition approach. Subsequently, it introduces video analysis techniques and dual signal fusion methods to capitalize on the strengths of both modalities, thereby enhancing the effectiveness of emotion recognition in online learning environments. The principal contributions of this paper are as follows:  
1. We propose a temporal convolutional network model equipped with channel attention, which processes PPG signals across multiple scales using CWT (continuous wavelet transform) and dynamically assigns weights to these scales using channel attention. This is followed by the extraction of spatiotemporal features using a TCN (temporal convolutional network) and GRUs (gated recurrent units). 2. We introduce a network model that fuses multi-scale spatiotemporal features, with a particular focus on key facial regions to counteract the effects of facial deformation, occlusion, and uneven lighting, thereby enhancing the utilization of spatial features. 3. We investigate a dual-modal data fusion method that combines video and PPG signals, and propose a model-level fusion strategy using an attention aggregation network with parallel connections. The remaining sections of this paper are arranged as follows: Section 2 surveys existing research on emotion recognition using PPG and video signal features, as well as a fusion method based on physiological signals and video measurement.