## 4 The Bassnet Model
### Learning Latent Representations to Encode the Relationship between \(X\) and \(Y\)
#### 4.4.2 Training Objective  
The left side of the VGAE--which is denoted as \(f_{\mu}(\cdot,\cdot)/f_{\sigma}(\cdot,\cdot)\), colored green in Figure 1--encodes \(x\) and \(y\) to the parameters \(\mu^{(x,y)}\) and \(\log\sigma^{2(x,y)}\) of a gaussian distribution \(\mathcal{N}^{(x,y)}\) over the latent space \(Z\). Conversely, given \(x\), the right side of the VGAE--denoted as \(g(\cdot,\cdot)\), colored purple--decodes a sample \(z\sim\mathcal{N}^{(x,y)}\) to a reconstruction of \(y\), being denoted \(\hat{y}\):  
\[\mu^{(x,y)} = f_{\mu}(x,y) \tag{1}\] \[\log\sigma^{2(x,y)} = f_{\sigma}(x,y)\] (2) \[z^{(x,y)} \sim \mathcal{N}^{(x,y)}\equiv\mathcal{N}(\mu^{(x,y)},\sigma^{(x,y)})\] (3) \[\hat{y}^{(x,y)} = g(x,z^{(x,y)}), \tag{4}\]  
where \(x,y\in R^{T\times C\times F}\) are CQT spectrogram windows, comprising \(T\) timesteps, \(C\) channels, and \(F\) frequency bins. Standard VAE Training Objective  
The standard VAE learning objective includes two terms in the loss function to be minimized: (1) the Kullback-Leibler divergence between the conditional distribution \(p(z|x)\) and the standard multivariate normal distribution \(\mathcal{N}(\mathbf{0},\mathbf{I})\) and (2) the negative log-likelihood of the data. Interpreting the real-valued decoded model output \(\hat{y}\) as the mean of an isotropic gaussian distribution \(p(x|z)\), which minimizes the negative log-likelihood of the data is equivalent to minimizing the mean squared error MSE between the data \(y\) and its reconstruction \(\hat{y}\)[64] [Section 6.2.2.1], so the objective can be written as:  
\[\mathcal{L}_{VAE} = D_{KL}(\mathcal{N}^{(x,y)},\;\mathcal{N}(\mathbf{0},\mathbf{I})) +MSE(y,\;\hat{y}^{(x,y)}). \tag{5}\]  
Enforcing Temporal Stability in Latent Representations  
Note that, in our current setting, where there is a temporal ordering on data instances (i.e., consecutive spectrogram windows), this learning objective does not pose any constraint on the evolution of \(z\) through time. Thus, \(z\) may be trained to encode aspects of the relation between \(x\) and  
Figure 3: Dilated residual block  
\(y\) that vary strongly through time. The intuition underlying slow feature analysis (Section 2) that the semantically relevant aspects of time series tend to remain stable over time suggests that the learning objective should be modified in order to enforce the temporal stability of \(z\). To this aim, we propose to modify/extend both the divergence and reconstruction loss terms in \(\mathcal{L}_{VAE}\) while using the notion of cross-reconstruction. Cross-reconstruction has been used in different settings, for example, for multimodal clustering [65], zero-shot learning [66], and learn transformation invariant representations [67]. In the current context, we implement cross-reconstruction as follows: For a given instance \((x,y)\), we select a second instance \((u,v)\) at random from the temporal vicinity of \((x,y)\). Using the equations, we compute the cross-reconstructions \(\hat{y}^{(u,v)}\) and \(\hat{v}^{(x,y)}\):  
\[\hat{y}^{(u,v)} = g(x,z^{(u,v)}) \tag{6}\] \[\hat{v}^{(x,y)} = g(u,z^{(x,y)}) \tag{7}\]  
Minimizing the cross-reconstruction loss terms \(MSE(y,\ \hat{y}^{(u,v)})\) and \(MSE(u,\ \hat{v}^{(x,y)})\) encourages the robustness of the decoding step against the variability of \(z\) within a temporal neighborhood, but it does not necessarily enforce the stationarity of \(z\) within the neighborhood. The latter can be achieved by minimizing what we call the cross-divergence loss: the Kullback-Leibler divergence between \(\mathcal{N}^{(x,y)}\) and \(\mathcal{N}^{(u,v)}\). Adding the cross-divergence loss and replacing the reconstruction loss term in Equation (5) by the cross-reconstruction loss terms, we obtain the following loss function for an instance \((x,y)\):  
\[ \begin{split}\mathcal{L}_{train}(x,y)&=D_{KL}( \mathcal{N}^{(x,y)},\ \mathcal{N}^{(u,v)})\\ &\quad+\ D_{KL}(\mathcal{N}^{(x,y)},\ \mathcal{N}(\mathbf{0}, \mathbf{I}))\\ &\quad+\ MSE(y,\ \hat{y}^{(u,v)})\end{split}