## 4 Sound-to-Image Translator
### Training Methodology  
Here we define a set of spectrogram-embedding-image data triples \(\{S_{i},\ x_{i},\ Y_{i}\}\), consisting of spectrograms \(S_{i}\in\mathbb{R}^{h\times w}\), audio feature vectors \(x_{i}\in\mathbb{R}^{f}\), and real-color images \(Y_{i}\in\mathbb{R}^{h\times w\times c}\), in which each element of the triple corresponds to the same \(i^{th}\) acoustic event. All the real numbers are limited to the interval \([-1,\ 1]\), since pixel values are normalized to the mentioned range before entering the network, and audio embeddings, as well as the realness-and-coherence score (RC-score) \(r\in\mathbb{R}\), both activated by TanH, fit the same interval. Regarding dimensions: \(h\) and \(w\) are, respectively, the height and width of spectrograms or images, \(c\) is the number of channels of color images, and \(f\) is the dimensionality of the audio feature space. The audio encoder \(A_{E}\) and the audio decoder \(A_{D}\) are defined, respectively, as the transformations \(A_{E}(S):\mathbb{R}^{h\times w}\rightarrow\mathbb{R}^{f}\) and \(A_{D}(x):\mathbb{R}^{f}\rightarrow\mathbb{R}^{h\times w}\). From the side of the adversarial networks, the generator \(G\) and the discriminator \(D\) are defined, respectively, as \(G(x):\mathbb{R}^{f}\rightarrow\mathbb{R}^{h\times w\times c}\) and \(D(Y\lor G(x),\ x):\mathbb{R}^{h\times w\times c+f}\rightarrow\mathbb{R}\), where \(Y\) denotes the real image that is entered into the discriminator, alternating with the synthetic image from \(G(x)\). With respect to the computation of loss functions, we chose to use the mean squared error (MSE) in the pixel space for measuring the distance between the target and the generated spectrogram/image during the training of both the audio autoencoder and the generator. This strategy provided an unbiased and scalable solution, enabling us to perform extensive testing, so that, by observing the behavior of the translator, we were able to improve the architecture and tune the networks for enhanced performance. The optimization of the autoencoder networks \(A_{E}\) and \(A_{D}\) is performed, minimizing the pixel loss \(L_{A}\big{(}S,\hat{S}\big{)}\) defined in Equation (1), where \(b\) is the batch size, and, as stated earlier, \(h\) and \(w\) are the height and width of spectrograms. The loss is computed as the MSE between the target spectrogram \(S\) and the generated one \(\hat{S}\gets A_{D}\big{(}A_{E}(S)\big{)}\). We include the batch iteration in the equations, since this is how the losses are effectively computed, being averaged among all instances of the batch. \[L_{A}\big{(}S,\ \hat{S}\big{)}=\frac{1}{bhw}\sum_{i=1}^{b}\sum_{j=1}^{h}\sum_{k= 1}^{w}\Big{(}S_{ijk}-\hat{S}_{ijk}\Big{)}^{2} \tag{1}\]  
The discriminator \(D\) is optimized through the minimization of the score loss \(L_{D}(r,\ \hat{r})\) defined in Equation (2), which is calculated from the batch-averaged MSE between the output RC-score \(\hat{r}\gets D(Y\lor G(x),x)\) and the target RC-score \(r\), which can be the maximum (1) or the minimum (\(-1\)) score value, depending on whether the input image is real or synthetic, respectively. \[L_{D}(r,\ \hat{r})=\frac{1}{b}\sum_{i=1}^{b}(r-\hat{r}_{i})^{2} \tag{2}\]  
The optimization of the generator \(G\) is guided by two objectives. The first aims to minimize the pixel loss \(L_{G}\big{(}Y,\hat{Y}\big{)}\) defined in Equation (3), where \(c\), as stated earlier, is the number of channels of color images. The loss is computed as the MSE between the target image \(Y\) and the generated one \(\hat{Y}\gets G(x)\). \[L_{G}\big{(}Y,\ \hat{Y}\big{)}=\frac{1}{bchw}\sum_{i=1}^{b}\sum_{j=1}^{c}\sum_{k= 1}^{h}\sum_{l=1}^{w}\Big{(}Y_{ijkl}-\hat{Y}_{ijkl}\Big{)}^{2} \tag{3}\]  
The second objective aims to minimize the adversarial loss based on the RC-score obtained from the discriminator \(D\).