## 1 Introduction  
Speech emotion recognition (SER) refers to extracting and analyzing emotion-related features from speech signals, allowing the computer to understand the speaker's emotional expression [1]. As an essential part of affective computing, SER is the key to facilitating natural human-computer interactions (HCIs) [2]. The HCI system with an affective computing module has been applied to many tasks, such as psychological assessment [3], mobile services [4], and safe driving [5]. Decades of research in SER have been devoted to modeling emotional representations from linguistic (e.g., lexical, syntactic, discourse and rhetorical features) and paralinguistic (e.g., supra-segmental phoneme and prosodic features) characteristics and to developing appropriate algorithms for implementing robust and effective emotion recognition [6; 7; 8; 9; 10]. Recently, significant progress has been made in the field of SER using multi-scale convolutional neural networks (MSCNNs). MSCNNs can extract multi-scale temporal and spatial features. Compared to single-scale network methods, MSCNNs [11; 12; 13; 14; 15] effectively capture emotion-related features of variable lengths from speech inputs and are expected to further enhance semantic understanding and emotion recognition. However, the MSCNN module in the previous research used a one-way transmission design. The temporal dynamic range that deep emotional features can retain depends on the size of the convolutional kernel. This unidirectional design can lead to the loss of temporal structure of speech in progressive resolution reduction. To enhance SER performance, it is crucial to learn both global-local semantic representation and temporal dynamics in speech. The main limitations comprise the following: (1) The high-level semantic features learned by the hierarchical MSCNN layers suffer from the loss of temporal structure of speech. (2) In context-aware semantic understanding, the MSCNN independently produces features that ignore the long-term relation between multi-scale sound units, and the correlations between multi-scale features should be explored. Based on the limitations of the existing feature extraction networks in emotional feature extraction, we propose a novel framework learning context-aware representation using a multi-scale feature pyramid network (MSFPN) for SER. We explore the bi-directional fusion of multi-scale semantic features using a feature pyramid network and preserve the resolution for temporal dynamics learning and global-local semantic understanding. More specifically, we adopt parallel MSCNN groups for multi-scale feature learning, supplemented with a forward fusion mechanism to fuse these features into high-level semantic features. Moreover, to learn the local-global correlations in MSCNN, this paper improves and uses the convolutional self-attention (CSA) [16] layer to focus on the emotion-related periods in the local region effectively. The MSFPN enhances the connections between adjacent elements and captures the interactions between features extracted by different attention heads. In the top-down pathway, features are rejoined into low-level acoustic features by backward fusion, and bi-directional long short-term memory (BiLSTM) is adopted to generate an utterance-level representation for emotion classification. Our research contributions can be summarized as follows:  
1. **We have conducted a detailed exploration of the application of feature pyramids in speech emotion recognition for the first time. We enhanced the MSCNN by integrating the CSA module to better capture local emotional correlations. **
2. **We have improved the CSA by using a multi-scale convolutional module, avoiding the degradation problem of the convolutional attention network. **
3. **We designed a backward fusion approach that effectively captures features across different levels of detail, successfully preserving the importance of local dynamics and deep semantics in emotional representation.