# Enhancing Embedded Space with Low-Level Features for Speech Emotion Recognition
## 2 Materials and Methods
### Audio Representations  
In the performed experiments, as base features (\(\gamma_{1}\)), we have used two time-frequency representations - CQT (Constant-Q Transform) and Mel spectrograms. This choice depends on previous research [18], where these representations obtained significantly higher results than the regular spectrogram in classifying emotional states using convolutional networks. Moreover, these representations present two different frequency scales. The frequency distribution is geometric for the constant-Q spectrogram and the ratio of the band's center frequency to its width is constant. As a result, the resulting frequency scale has a different accuracy reproduction in the low and medium frequency range compared to Mel scale [19]. Figure 1: The proposed architecture of the injection mechanism. ```
Input: Audio representations \(\{R_{1}^{i},R_{2}^{i}\}_{i=1}^{N}\) Output: Predicted labels \(\{Y^{i}\}_{i=1}^{N}\) \(Y=\{\}\) for\(i=1\) to N do  Input \(r_{1}^{i}\) into neural network and next to \(FC_{1}\) for mapping to specific size and get feature map \(x_{1}^{i}\). Send \(r_{2}^{i}\) to \(norm\) layer to get vector \(x_{2}^{i}\). Concatenate \(x_{1}^{i}\) and \(x_{2}^{i}\) to get final feature vector \(X^{i}\). Send features \(X^{i}\) to final classification layer \(FC_{2}\), execute \(softmax\) operation and get predicted label \(y^{i}\). \(Y^{i}\gets y^{i}\) endfor
```  
**Algorithm 1** Injection mechanism  
Mel spectrogram is calculated based on the spectrogram, which is obtained using the STFT (Short-Time Fourier Transform) algorithm, and then, the frequencies are converted to the Mel scale according to Formula (1):  
\[f_{mel}(f_{hz})=2595\log_{10}\left(1+\frac{f_{hz}}{700}\right), \tag{1}\]  
where \(f_{mel}\) is the Mel frequency and \(f_{hz}\) is the frequency in Hertz. In turn, in CQT, compared to a standard spectrogram, each signal frame is transformed using the Constant-Q Transform [20] method. This method uses a bank of filters corresponding to subsequent octaves. The central frequencies of successive filters are determined according to Formula (2). \[f_{m}=\left(2^{\frac{1}{b}}\right)^{m}\cdot f_{min}, \tag{2}\]  
where \(f_{m}\) is the center frequency of the \(m\)-th filter, \(b\) is the number of filters per octave and \(f_{min}\) is the minimum frequency. The constant \(Q\) is then calculated according to the Equation (3). \[Q=\frac{f_{m}}{\Delta f_{m}}=\frac{f_{m}}{f_{m+1}-f_{m}}=\left(2^{\frac{1}{b}}- 1\right)^{-1}. \tag{3}\]  
In the final step, the frequency domain representation is computed for subsequent signal frames according to the Formula (4). \[X_{p}[m]=\frac{1}{N[m]}\cdot\sum_{n=0}^{N[m]-1}W[m,n]\cdot x_{p}[n]\cdot e^{ \frac{-2n\Omega n}{N[m]}}, \tag{4}\]  
where \(W\) is the window function, \(p\) is the signal frame number, and \(N[m]\) is the frame length for bin \(m\), which is determined according to the following equation:  
\[N[m]=Q\cdot\frac{f_{s}}{f_{m}}, \tag{5}\]  
where \(f_{s}\) denotes sampling rate. This representation was initially proposed for reproducing the Western musical scale [21], but it can also be used to classify emotional states based on speech signals [18; 22]. As a set of injected low-level features (\(\gamma_{2}\)), we have used a set of statistical features calculated based on the fundamental frequency signal \(F_{0}\), three consecutive formants: \(F_{1}\), \(F_{2}\), \(F_{3}\) and derivatives of these values, resulting in 99 features in total. Our motivation for selecting the fundamental frequency and its harmonics is based on numerous studies [16; 23; 24; 25], that confirm the crucial role of \(F_{0}\) in emotion recognition from speech signals. The fundamental frequency and its harmonics are strongly linked to speech intonation, which is an important element in expressing emotions. Moreover, in our work, we also considered previous studies focused on the interpretation of convolutional networks in the context of emotion recognition [26].