# Multi-Level Attention-Based Categorical Emotion Recognition Using Modulation-Filtered Cochleagram
## 2 Modulation-Filtered Cochleagram
### Modulation-Filtered Cochleagram Features  
The modulation-filtered cochleagram feature is used to capture the temporal modulation cues from emotional speech and achieves significant effects in dimensional emotion prediction. In this study, we explore the potential of employing the modulation-filtered cochleagram features for categorical emotion recognition. The emotional speech signal \(s(t)\) is first filtered using a bank of Gammatone cochlea filters. Then, the temporal envelope of the subchannel signal is extracted using Hilbert transform. Furthermore, the \(m\) th modulation filter in the \(n\)th channel envelope signal is used to obtain the spectral-temporal modulation signal \(s_{mu}(n,m,i)\), which is defined as:  
\[s_{mu}(n,m,i)=\mathrm{w}(t_{w})\cdot s_{m}(n,m,(i-1)\cdot Len_{s}+t_{w}) \tag{1}\]  
where \(\mathrm{w}(t_{w})\) is the window function, \(t_{w}\) is the time window size, and \(Len_{s}\) is the frame shift. \(s_{mu}(n,m,i)\) refers to the \(m\) th modulation channel and the \(n\) th cochlea acoustic channel of the \(i\) th modulation unit, and a total of \(n*m\) channel signals are generated, where \(1\leq i\leq L\), \(L\) is equal to \(Len_{t}/Len_{s}\), and \(Len_{t}\) is the total length of the speech signal \(s(t)\). \(s_{m}(n,m,(i-1)\cdot Len_{s}+t_{w})\) is the spectral-temporal modulation signal of the \(n\) subchannel and the \(m\) subchannel of the \(i\) modulation unit. \(s_{mu}(n,m,i)\) represents the m modulation subchannel in the \(n\) acoustic subchannel. The calculation formula is as follows:  
\[s_{m}(n,m,t)=m_{t}(m,t)*s_{\mathrm{e}}(n,t),1\leq m\leq M \tag{2}\]  
where \(m_{t}(m,t)\) is the pulse response of the modulation filterbank, \(M\) is the number of channels in the modulation filterbank, and \(s_{\mathrm{e}}(n,t)\) is calculated by \(s_{\mathrm{g}}(n,t)\) as the size of the complex resolution signal \(s_{\mathrm{g}}^{\prime}(n,t)=s_{\mathrm{g}}(n,t)+\mathcal{H}\left\{s_{ \mathrm{g}}(n,t)\right\}\). \(\mathcal{H}\left\{\cdot\right\}\) represents the Hilbert transformation. Therefore, \(s_{\mathrm{e}}(n,t)\) is calculated as follows:  
\[s_{\mathrm{e}}(n,t)=\left|s_{\mathrm{g}}(n,t)\right|=\sqrt{s_{\mathrm{g}}(n, t)^{2}+\mathcal{H}\left\{s_{\mathrm{g}}(n,t)\right\}^{2}} \tag{3}\]  
The \(s_{\mathrm{g}}(n,t)\) represents the speech signal \(s(t)\) of the \(n\)th channel of the speech signal processed by the auditory filter, using the following formula:  
\[s_{\mathrm{g}}(n,t)=g_{t}(n,t)*s(t),1\leq n\leq N \tag{4}\]  
where \(g_{t}(n,t)\) represents the pulse response of the \(n\)th channel of the filterbank, \(*\) represents the convolution operation, \(t\) is the number of samples in the time domain, and \(N\) is the number of channels in the auditory filterbank. The Gammatone filterbank is used to simulate the motion of the cochlear basilar membrane, and its pulse response is the product of the Gamma distribution and the cosine signal:  
\[g_{t}(n,t)=\mathrm{A}^{n_{t}-1}\mathrm{exp}\left(-2\pi w_{f}ERB_{N}(f_{n})t \right)\mathrm{cos}(2\pi f_{n}t+\varphi) \tag{5}\]  
where \(A,n_{f}\text{ and }w_{f}\) are the amplitude, order, and bandwidth of the filter, \(\mathrm{A}t^{n_{t}-1}\mathrm{exp}\left(-2\pi w_{f}ERB_{N}(f_{n})t\right)\) is the amplitude term of the Gamma distribution representation, \(f_{n}\) is the central frequency of the \(n\)th channel of the filter, and \(ERB_{N}(f_{n})\) is the equivalent rectangular bandwidth of \(f_{n}\), which is a psychoacoustic measure of the width of the auditory filter at each point along the cochlea. The calculation formula is provided as follows:  
\[ERB_{n}(f_{n})=\frac{f_{n}}{Q_{ear}}+B_{min} \tag{6}\]  
where \(f_{n}\) is the central frequency of the n th filter, \(\frac{f_{n}}{Q_{ear}}\) is the quality factor, which approximates the filtering quality of the high-frequency band, and \(B_{min}\) is the minimum bandwidth, representing the approximation of the filtering quality of the low-frequency band. \(Q_{ear}\) and \(B_{min}\) generally adopt the values proposed in the literature [35], with 9.26449 and 24.7, respectively.