# Disentangled Feature Learning for Noise-Invariant Speech Enhancement
## 3 Proposed Method
### Adversarial Training for Disentangled Features  
Neural network parameters are optimized by using the objective function given in Equation (8) via adversarial learning. \(D_{s}\) and \(D_{n}\) are trained to minimize \(\mathcal{L}_{D_{s}}\) and \(\mathcal{L}_{D_{n}}\), and \(DE_{n}\) and \(DE_{s}\) are also trained to minimize \(\mathcal{L}_{DE_{n}}\) and \(\mathcal{L}_{DE_{s}}\). As for the optimization of \(E\), it was essential to ensure that it should produce disentangled features. This idea was implemented by minimizing \(\mathcal{L}_{D_{s}}\) and \(\mathcal{L}_{D_{n}}\) while maximizing \(\mathcal{L}_{DE_{n}}\) and \(\mathcal{L}_{DE_{s}}\) in an adversarial manner with respect to the encoder parameter \(\theta_{E}\). Such a mini-max competition eventually converges to the point where the encoder network generates the noise-confusing latent feature \(\mathbf{z}_{s}\) and the speech-confusing latent feature \(\mathbf{z}_{n}\), disentangled in the latent feature space. \(D_{s}\) and \(D_{n}\) then use \(\mathbf{z}_{s}\) and \(\mathbf{z}_{n}\) as input respectively and produce noise-invariant speech \(\mathbf{\hat{s}}\). In summary, optimizations of the network parameters are given by  
\[\begin{split}(\hat{\theta}_{E},\hat{\theta}_{D_{s}},\hat{\theta} _{D_{n}})&=\operatorname*{arg\,min}_{\theta_{E}\hat{\theta}_{D_{s }}\theta_{D_{n}}}\mathcal{L}_{T}(\theta_{E},\theta_{D_{s}},\theta_{DE_{n}}, \hat{\theta}_{DE_{s}}),\\ (\hat{\theta}_{DE_{n}},\hat{\theta}_{DE_{s}})&= \operatorname*{arg\,max}_{\theta_{DE_{n}}\theta_{DE_{s}}}\mathcal{L}_{T}( \theta_{E},\theta_{D_{s}},\hat{\theta}_{D_{n}},\theta_{DE_{s}},\theta_{DE_{s }}),\end{split} \tag{9}\]  
where \(\hat{\theta}_{(\cdot)}\) denotes the optimal parameters for each given network \((\cdot)\).  
The network parameters defined by Equation (9) can be found as a stationary point of the following gradient updates:  
\[\begin{split}\theta_{E}&\longleftarrow\theta_{E} &-\mu(\frac{\partial\mathcal{L}_{D_{s}}}{\partial\theta_{E}}+ \alpha\frac{\partial\mathcal{L}_{D_{n}}}{\partial\theta_{E}}-\lambda_{1} \frac{\partial\mathcal{L}_{DE_{n}}}{\partial\theta_{E}}-\alpha\lambda_{2}\frac {\partial\mathcal{L}_{DE_{s}}}{\partial\theta_{E}}),\\ \theta_{D_{s}}&\longleftarrow\theta_{D_{s}}&-\mu \frac{\partial\mathcal{L}_{D_{s}}}{\partial\theta_{D_{s}}},\\ \theta_{D_{n}}&\longleftarrow\theta_{D_{n}}&-\mu \frac{\partial\mathcal{L}_{D_{n}}}{\partial\theta_{D_{n}}},\\ \theta_{DE_{n}}&\longleftarrow\theta_{DE_{n}}&-\mu \frac{\partial\mathcal{L}_{DE_{s}}}{\partial\theta_{DE_{s}}},\\ \theta_{DE_{s}}&\longleftarrow\theta_{DE_{s}}&-\mu \frac{\partial\mathcal{L}_{DE_{s}}}{\partial\theta_{DE_{s}}},\\ \end{split} \tag{10}\]  
where \(\mu\) indicates the learning rate. The updates of Equation (10) are very similar to stochastic gradient descent (SGD) updates for the feed-forward deep model that comprises the encoder fed into the decoders and into the disentanglers. The difference was that the gradients from the decoders and disentanglers were subtracted with loss weighted by \(\lambda_{1}\), \(\lambda_{2}\), and \(\alpha\), instead of being summed. The negative coefficient \(-\lambda_{1}\) and \(-\lambda_{2}\) enable the encoder to induce the maximization of \(\mathcal{L}_{DE_{s}}\) and by reversing the gradients during the back-propagation. If both \(\lambda_{1}\) and \(\lambda_{2}\) were set to zero, the neural network structure presented in Figure 1 became equivalent to the conventional DNN structure. The optimized networks \(E\), \(D_{s}\) and \(D_{n}\) were then used during the test stage for generating the clean speech estimates given the noisy test speech samples.