# Disentangled Feature Learning for Noise-Invariant Speech Enhancement
## 3 Proposed Method
### Neural Network Architecture  
Our neural network consists of five sub-networks: (i) an encoder (\(E\)), (ii) a speech decoder (\(D_{s}\)), (iii) a noise decoder (\(D_{n}\)), (iv) a noise disentangler (\(DE_{n}\)), and (v) a speech disentangler (\(DE_{s}\)). The overall architecture of the proposed model is illustrated in Figure 1.  
We extracted the magnitude spectra as the raw features of all signal components. Only the magnitude spectra were estimated while the phase parts of the noisy speech are kept intact. Let us denote the magnitude spectra of the noisy speech, clean speech, and noise as \(\mathbf{x}\in\mathbb{R}^{F\times(2\tau+1)}\), \(\mathbf{s}\in\mathbb{R}^{F\times 1}\), and \(\mathbf{n}\in\mathbb{R}^{F\times 1}\), respectively, where \(F\) denotes the number of frequency bins and \(\tau\) represents an input context expansion parameter (i.e., one current frame, \(\tau\) previous and \(\tau\) next frames). The encoder \(E\) learned a function that maps \(\mathbf{x}\) into speech and noise latent features, defined by the neural network parameter \(\theta_{E}\) as follows:  
\[(\mathbf{z}_{s},\mathbf{z}_{n})=E(\mathbf{x};\theta_{E}), \tag{1}\]  
where \(\mathbf{z}_{s}\in\mathbb{R}^{M\times 1}\) and \(\mathbf{z}_{n}\in\mathbb{R}^{M\times 1}\) indicate \(M\)-dimensional speech and noise latent features, respectively. Similarly, \(D_{s}\) and \(D_{n}\) learn mappings parameterized by \(\theta_{D_{s}}\) and \(\theta_{D_{n}}\), respectively, as follows:  
\[\mathbf{\hat{m}}_{s} =D_{s}(\mathbf{z}_{s};\theta_{D_{s}}), \tag{2}\] \[\mathbf{\hat{m}}_{n} =D_{n}(\mathbf{z}_{n};\theta_{D_{n}}),\]  
Figure 1: The architecture of the proposed model for disentangled feature learning. where \(\hat{\mathbf{m}}_{s}\in\mathbb{R}^{F\times 1}\) and \(\hat{\mathbf{m}}_{n}\in\mathbb{R}^{F\times 1}\) denote the estimated speech and noise masks, respectively. The time-frequency masks were constrained such that the sum of the estimated values should be equal to the input noisy speech. Given the masks from both of the decoders, we can obtain the predicted speech and noise through a deterministic layer [22]. Given \(\hat{\mathbf{m}}_{s}\) and \(\hat{\mathbf{m}}_{n}\), the predicted magnitude spectra of speech \(\hat{\mathbf{s}}\in\mathbb{R}^{F\times 1}\) and noise \(\hat{\mathbf{n}}\in\mathbb{R}^{F\times 1}\) can be calculated as  
\[\hat{\mathbf{s}} =\frac{\hat{\mathbf{m}}_{s}}{\hat{\mathbf{m}}_{s}+\hat{\mathbf{m }}_{n}}\otimes\mathbf{x}, \tag{3}\] \[\hat{\mathbf{n}} =\frac{\hat{\mathbf{m}}_{n}}{\hat{\mathbf{m}}_{s}+\hat{\mathbf{m }}_{n}}\otimes\mathbf{x},\]  
where the addition, division, and product (\(\otimes\)) operators were executed element-wise. Finally, \(DE_{n}\) and \(DE_{s}\) were trained to separate the noise attributes from the speech latent features, and vice versa. \(DE_{n}\) and \(DE_{s}\) are respectively parameterized by \(\theta_{DE_{n}}\) and \(\theta_{DE_{s}}\) as follows:  
\[\hat{\mathbf{n}} =DE_{n}(\mathbf{z}_{s};\theta_{DE_{n}}), \tag{4}\] \[\tilde{\mathbf{s}} =DE_{s}(\mathbf{z}_{n};\theta_{DE_{s}}),\]  
where \(\tilde{\mathbf{s}}\in\mathbb{R}^{F\times 1}\) and \(\tilde{\mathbf{n}}\in\mathbb{R}^{F\times 1}\) represent the speech and noise components, respectively estimated from the latent features. Note that \(\tilde{\mathbf{s}}\) and \(\tilde{\mathbf{n}}\) differ from \(\hat{\mathbf{s}}\) and \(\hat{\mathbf{n}}\) in Equation (3). \(\hat{\mathbf{s}}\) and \(\tilde{\mathbf{n}}\) were generated by the disentanglers which were trained to make the encoder difficult to predict the speech and noise. The GRLs are inserted between the encoder and the disentanglers to establish an adversarial setting. On the other hand, \(\hat{\mathbf{s}}\) and \(\hat{\mathbf{n}}\) were well estimated by the corresponding decoders.