## 3 Learned Separation
### Model Fitting
#### 3.2.2 Policy Gradients  
We use a neural network with network parameters \(\theta\) both to give the _policy_\(\pi_{\theta}(\varpi_{\mathrm{s}}|Y)\), which is the (discrete or continuous) probability density for the stochastic parameters \(\varpi_{\mathrm{s}}\) given the input spectrum \(Y\) and also to compute the deterministic parameters \(\varpi_{\mathrm{d}}\) from the input and the stochastic parameters. With this, we can express the _loss function_ as:  
\[L(\varpi_{\mathrm{s}},\theta,D,Y)\coloneqq L\big{(}\varpi_{\mathrm{s}},\varpi _{\mathrm{d}}(\varpi_{\mathrm{s}},\theta,Y),D,Y\big{)}=d_{2,\delta}^{\mathrm{ d,abs}}\big{(}Y,y(\varpi_{\mathrm{s}},\varpi_{\mathrm{d}},D)\big{)}=d_{2, \delta}^{\mathrm{d,abs}}(Y,y). \tag{14}\]  
For the expected loss, we follow the usual computation, cf. [2], but also apply the product rule for the deterministic parameters:  
\[\
abla_{\theta}\,\mathrm{E}_{\pi_{\theta}(\varpi_{\mathrm{s}}|Y) }\big{[}L(\varpi_{\mathrm{s}},\theta,D,Y)\big{]} \tag{15}\] \[=\
abla_{\theta}\int\pi_{\theta}(\varpi_{\mathrm{s}}|Y)\,L( \varpi_{\mathrm{s}},\theta,D,Y)\;\mathrm{d}\varpi_{\mathrm{s}}\] \[=\int\
abla_{\theta}\pi_{\theta}(\varpi_{\mathrm{s}}|Y)\,L( \varpi_{\mathrm{s}},\theta,D,Y)+\pi_{\theta}(\varpi_{\mathrm{s}}|Y)\,\
abla_{ \theta}L(\varpi_{\mathrm{s}},\theta,D,Y)\;\mathrm{d}\varpi_{\mathrm{s}}\] \[=\int\pi_{\theta}(\varpi_{\mathrm{s}}|Y)\,\bigg{(}\frac{\
abla_{ \theta}\pi_{\theta}(\varpi_{\mathrm{s}}|Y)}{\pi_{\theta}(\varpi_{\mathrm{s}} |Y)}\,L(\varpi_{\mathrm{s}},\theta,D,Y)+\
abla_{\theta}L(\varpi_{\mathrm{s}},\theta,D,Y)\bigg{)}\;\mathrm{d}\varpi_{\mathrm{s}}\] \[=\mathrm{E}_{\pi_{\theta}(\varpi_{\mathrm{s}}|Y)}\big{[}\
abla_{ \theta}\log\pi_{\theta}(\varpi_{\mathrm{s}}|Y)\,L(\varpi_{\mathrm{s}},\theta, D,Y)+\
abla_{\theta}L(\varpi_{\mathrm{s}},\theta,D,Y)\big{]},\]  
where we refer to the first term in the sum as the _policy gradient_ and on the second term as the _backpropagation gradient_. The applicability of the Leibniz integral rule can be shown under realistic conditions.  
Figure 2: Probability density functions of gamma distributions for different parameter choices. For \(a^{\Gamma}\geq 1\), the function has the mode at \((a^{\Gamma}-1)/\beta^{\Gamma}\), while for \(a^{\Gamma}<1\), the function tends to \(\infty\) at zero. In our proposed method, the network selects a distribution shape for each inharmonicity coefficient \(b_{j}\) by its outputs \(a_{j}^{\Gamma},\beta_{j}^{\Gamma}\).  
The total dimensionality of \(\varpi_{\rm s}\) is too high to represent \(\pi_{\theta}(\varpi_{\rm s}|Y)\) as a whole. Therefore, we decompose the log-probability density into:  
\[\log\pi_{\theta}(\varpi_{\rm s}|Y)\] \[=\log\bigl{(}\pi_{\theta}(\varpi_{\rm s,1}|Y)\cdot\pi_{\theta}( \varpi_{\rm s,2}|Y,\varpi_{\rm s,1})\cdot\cdots\pi_{\theta}(\varpi_{\rm s,m}|Y, \varpi_{\rm s,1},\ldots,\varpi_{\rm s,m-1})\bigr{)} \tag{16}\] \[=\log\pi_{\theta}(\varpi_{\rm s,1}|Y)+\log\pi_{\theta}(\varpi_{ \rm s,2}|Y,\varpi_{\rm s,1})+\ldots+\log\pi_{\theta}(\varpi_{\rm s,m}|Y,\varpi _{\rm s,1},\ldots,\varpi_{\rm s,m-1}).\]  
In practice, we first sample the stochastic parameters \(\varpi_{\rm s,1}\) for the first tone, then obtain the deterministic parameters \(\varpi_{\rm d,1}\), and from these compute a model spectrum \(y_{1}\). For each additional tone \(j=2,\ldots,m\), we gain \(\varpi_{\rm s,j}\) and \(\varpi_{\rm d,j}\) depending on the previous spectra \(y_{1},\ldots,y_{j-1}\), which encapsulate all the relevant information about the parameters of the previous tones.  
The parameters \(\
u_{j},\eta_{j}\) are sampled via a joint categorical distribution, making sure that no instrument plays more than one tone. For each possible value of \((\
u_{j},\eta_{j})\), the network gives a value for the deterministic parameters \(\varpi_{\rm d,j}\) as well as for the parameters \(\alpha_{j}^{\Gamma},\beta_{j}^{\Gamma}\) for the gamma distribution for \(b_{j}\).