## 3 Learned Separation
### Sampling for Gradient Estimation  
In order to apply a gradient descent method, we need to compute the expectation in (15). However, the set of possible parameters \(\varpi_{\text{s}}\) is much too large to do so analytically, so we have to estimate it instead. For this, we use:  
\[\hat{g}_{\pi_{0},Y}=\frac{1}{S}\sum_{i=1}^{S}\Bigl{(}\
abla_{\theta}\log\pi_{ \theta}(\varpi_{\text{s}}^{i}|Y)\cdot\bigl{(}L(\varpi_{\text{s}}^{i},\theta, D,Y)-C(\theta,D,Y)\bigr{)}+\
abla_{\theta}L(\varpi_{\text{s}}^{i},\theta,D,Y) \Bigr{)}, \tag{22}\]  
with \(\varpi_{\text{s}}^{1},\dots,\varpi_{\text{s}}^{S}\sim\pi_{\theta}(\varpi_{ \text{s}}|Y)\), where \(S\in\mathbb{N}\) is the number of samples and \(C(\theta,D,Y)\) is the _baseline_, cf. [2]. The baseline plays a crucial role in reducing the variance of the gradient, and a common choice is:  
\[C(\theta,D,Y)=\operatorname{E}_{\pi_{\theta}(\varpi_{\text{s}}|Y)}\bigl{[}L( \varpi_{\text{s}},\theta,D,Y)\bigr{]}. \tag{23}\]  
As long as the baseline is independent of the samples \(\varpi_{\text{s}}^{1},\dots,\varpi_{\text{s}}^{S}\), the estimator (22) is unbiased. However, the easiest way to estimate the baseline is via:  
\[\hat{C}(\theta,D,Y)=\frac{1}{S}\sum_{i=1}^{S}L(\varpi_{\text{s}}^{i},\theta, D,Y), \tag{24}\]  
which obviously depends on the samples and therefore introduces a bias factor of \((S-1)/S\) for the policy gradient. If desired, one could correct for this bias by dividing by this factor, but it is generally advisable to be conservative about the policy gradient, so we choose to leave it uncorrected. However, the value for \(S\) should be chosen large enough for (24) to be a good estimator. During the training, it is expected that the policy will become increasingly deterministic, so less exploration will be performed. The problem with this is that multiple parameters need to be trained, and it is easy for the training process to get stuck in local minima. Therefore, we should encourage the exploration of more parameter choices even when the policy has stabilized. Inspired by the algorithm from [31], we thus define:  
\[\pi_{\theta}^{r_{j}}(\varpi_{\mathrm{s},j}|Y)\coloneqq\frac{\pi_{\theta}(\varpi _{\mathrm{s},j}|Y)^{r_{j}}}{\int\pi_{\theta}(\varpi_{\mathrm{s},j}|Y)^{r_{j}} \,\mathrm{d}\varpi_{\mathrm{s},j}},\qquad r_{j}>0. \tag{25}\]  
The \(r_{j}\) parameter can be chosen separately for each tone; with \(R=(r_{1},\ldots,r_{m})\), we further define:  
\[\pi_{\theta}^{R}(\varpi_{\mathrm{s}}|Y)\coloneqq\pi_{\theta}^{r_{1}}(\varpi_{ \mathrm{s},1}|Y)\cdot\pi_{\theta}^{r_{2}}(\varpi_{\mathrm{s},2}|Y,\varpi_{ \mathrm{s},1})\cdot\cdot\pi_{\theta}^{r_{m}}(\varpi_{\mathrm{s},m}|Y,\varpi_ {\mathrm{s},1},\ldots,\varpi_{\mathrm{s},m-1}). \tag{26}\]  
We set \(S=3^{m}\) and let \(R_{1},\ldots,R_{S}\) be all the combinations of \(m\) elements out of the set \(\{1,0.1,0.01\}\), which turn out to be reasonable magnitudes. Additionally, we do not only accept the bias which causes an underestimation of the policy gradient, but we also artificially scale it down by a factor of 10. Since the policy gradient and the backpropagation gradient model different parameters, this is conceptually not a problem; however, if the gradient is scaled _too_ much, the interdependencies of the values in the network can then cause instabilities in the output of the stochastic parameters. Our modified gradient estimator is:  
\[\hat{g}_{\theta,Y}=\frac{1}{S}\sum_{i=1}^{S}\Bigl{(}\frac{1}{10}\
abla_{\theta }\log\pi_{\theta}^{R_{i}}(\varpi_{\mathrm{s}}^{i}|Y)\cdot\bigl{(}L(\varpi_{ \mathrm{s}}^{i},\theta,D,Y)-\hat{C}(\theta,D,Y)\bigr{)}+\
abla_{\theta}L( \varpi_{\mathrm{s}}^{i},\theta,D,Y)\Bigr{)}, \tag{27}\]  
where each tone parameter set \(\varpi_{\mathrm{s},j}^{i}\) is sampled according to the value of \(r_{j}\) inside \(R_{i}\). This includes the sparsity parameter \(u_{j}\), but the inharmonicity \(b_{j}\) is exempt from this modification altogether and always sampled according to \(\pi_{\theta}\).