## 3 Learned Separation
### Network Architecture  
We use a U-Net architecture with seven downsampling/upsampling steps; strides of \(4,4,4,4,4,4,3,2\); and \(80,160,\ldots,560\) one-dimensional filters with a size of 5. Finally, we add two more convolutional layers with 80 filters of sizes 3 and 1, respectively, before the linear output layer. All of the hidden layers have ReLU activation. The first hidden layer is a _CoordConv_ layer [37], which means that it is supplied with a linear range from 0.01 to 0 as an additional input channel. These design parameters were obtained via manual experimentation.  
For the first tone, the network is supplied with the input spectrum \(Y\) and the absolute value spectrum \(|Y|\). From this, the spectra \(y_{1}\) and \(y_{1}^{\text{dir}}\) are computed. For each following tone \(j=2,\ldots,m\), the network receives the residuals \(Y-y_{1}-\ldots-y_{j-1}\) and \(Y-y_{1}^{\text{dir}}-\ldots-y_{j-1}^{\text{dir}}\) along with their absolute values as well as the computed tone spectra \(y_{1},\ldots,y_{j-1}\) and \(y_{1}^{\text{dir}},\ldots,y_{j-1}^{\text{dir}}\). From these values, it then yields the spectra \(y_{j}\) and \(y_{j}^{\text{dir}}\). For those input components of the network that do not yet receive a spectrum for a particular tone, a constant 0 vector is given as input to the network instead. The data flow is illustrated in Figure 5.  
Since the amplitudes \((a_{j})\) are supposed to be non-negative, we apply the absolute value function to the respective output components of the network. The widths \((\sigma_{j})\) are kept positive via softplus, and they are clipped such that the value does not get too close to 0. For the continuous frequency offsets \((\tilde{v_{j}})\), a tanh function is used to keep them inside the interval \((-5,5)\). The positive parameters \((a_{j}^{\Gamma},\beta_{j}^{\Gamma})\) are obtained after applying the exponential function, and the probabilities for the Bernoulli distribution for the sparsity parameters \((u_{j})\) are mapped into the interval \((0,1)\) via a sigmoid function. The joint categorical distribution for the discrete frequencies \((\
u_{j})\) and instrument indices \((\eta_{j})\) is given in vectorial form as non-normalized log-probabilities, so we apply the _softmax_ mapping in order to obtain a valid discrete distribution. Doing this, we have to make sure that each instrument can only play one tone at a time by excluding that instruments that have already been assigned a tone from the sampling. For each parameter output by the network, we add a trainable scaling layer.  
In order to protect against potentially degenerate network output or gradients in the case of zeros in the input vector, a minimal amount of Gaussian noise is always added to \(Y\) prior to prediction, on a level that is negligable for any normal audio signals.  
Figure 5: Sampling architecture.  
The dictionary entries are supposed to be elements of the interval \([0,1]\). Non-negativity is usually satisfied automatically due to \(a_{j}\geq 0\), and for the upper bound, we add the loss \(\frac{1}{N_{\text{ins}}}\sum_{\eta}(\log(\max_{h}D[h,\eta]))^{2}\) to the training of the dictionary.