# Employing Huber and TAP Losses to Improve Inter-SubNet in Speech Enhancement
## 3 Inter-SubNet with New Loss Function
### TAP Loss  
Conventional loss functions such as MSE loss employed in the training of Inter-SubNet only take into account the differences between individual points of signals or signals' masks in either the time domain or frequency domain. Consequently, they have low correlations with perceptual speech quality or they may overemphasize high-energy regions in signals. To solve this problem, we can further improve deep networks by taking into account acoustic characteristics. This helps improve the perceptual quality of enhanced speech in addition to noise reduction. The research in [34] focuses on analyzing 25 acoustic features at each time frame of an input speech, which are thus called temporal acoustic parameters (TAPs). A differentiable estimator was created for these TAPs. Leveraging the differentiability of the TAP estimator, the built TAP loss can be employed to learn any deep network for speech enhancement.  
The four classes that cover the aforesaid 25 TAPs are as follows [34]  
* **Frequency-related**: These parameters capture characteristics related to the frequency content of the speech signal.
* **Energy- or amplitude-related**: These parameters reflect the loudness and energy levels of the speech signal, critical for understanding the intensity and dynamics of speech.
* **Spectral-balance-related**: These parameters assess the energy distribution across different frequency bands, providing insights into the tonal balance and clarity of the speech.
* **Other temporal parameters**: These parameters capture the temporal dynamics of the speech signal, including variations over time that are important for conveying speech rhythm and flow.  
Here, we introduce the derivation of TAP loss by referring to [34]. Let \(\mathbf{A_{y}}\) denote the TAPs of a time-domain utterance \(\mathbf{y}\). The TAP estimator produces an estimate of \(\mathbf{A_{y}}\) as  
\[\mathbf{\hat{A}_{y}}=\mathcal{TAP}(\mathbf{y}), \tag{8}\]  
where \(\mathcal{TAP}(\dot{)}\) denotes the TAP estimator operation, which converts \(\mathbf{y}\) to its complex spectrogram \(\mathbf{Y}\) and then passes \(\mathbf{Y}\) to a recurrent neural network to output the estimate \(\mathbf{\hat{A}_{y}}\). In order to learn the network parameters of the TAP estimator, the MAE is employed:  
\[\text{MAE}\big{(}\mathbf{A_{y}},\mathbf{\hat{A}_{y}}\big{)}=\mathbb{E}_{t,p} (|\mathbf{A_{y}}(t,p)-\mathbf{A_{\tilde{y}}}(t,p)|), \tag{9}\]  
where \(t\) and \(p\) are the time frame index and TAP index, respectively, and \(\mathbb{E}_{t,p}\) is the expectation operator over all time frames and TAPs.  
Once the aforesaid TAP estimator is learned, it is frozen as a fixed processing component to build TAP loss between the enhanced signal \(\mathbf{s}\) and the ground-truth signal \(\mathbf{\hat{s}}\).  
Let \(\mathbf{S}=\{S(t,f)\}\) be the short-time Fourier transform (STFT) of \(\mathbf{s}\); the frame-wise energy weights \(\mathbf{\omega}\) are calculated by  
\[\mathbf{\omega}=\frac{1}{F}\sum_{f=0}^{F-1}\|S(t,f)\|^{2}\in\mathbb{R}^{T}. \tag{10}\]  
Then, the final TAP loss is defined as  
\[\mathcal{L}_{\text{TAP}}(\mathbf{s},\hat{\mathbf{s}})=\text{MAE}(\mathcal{TAP }(\mathbf{s})\odot\sigma(\mathbf{\omega}),\mathcal{TAP}(\hat{\mathbf{s}})\odot \sigma(\mathbf{\omega})), \tag{11}\]  
where \(\sigma(.)\) is the sigmoid function, and \"\(\odot\)\" is the operation of element-wise multiplication with broadcasting. From Equation (11), TAP loss is in fact the MAE between ground-truth (clean) and estimated TAPs with smoothed frame energy-weighting. The smoothed frame energy weights \(\sigma(\mathbf{\omega})\) are used to highlight the TAPs of high-energy frames.