## 3 Inherent Challenges in S2I Translation Processes
### Addressing the Problem  
We train our translator without any type of supervision and using 48,945 sound-image pairs extracted from more than 8000 different scenes with diverse audiovisual content. These scenes are representative of the type of visual output we expect the translator to generate. Additionally, we test the translator exclusively with scenes unknown to the model. These criteria set our study apart from previous works that adopted more simplified approaches, without addressing content diversity and/or employing supervision/self-supervision for training, as explained in the previous section. In some of the studies mentioned, the task being performed is similar to an image retrieval process, in which the system fetches from a database the image that best fits the query. The exploration we have conducted, along with the corresponding solutions and the S2I translator we provide, represent a modest attempt to tackle this significant challenge. Potential alternative approaches include attention-based methods [50; 51], causal reasoning [52; 53], explicit incorporation of prior knowledge through rules and constraints [54], bags of acoustic events [55] and/or visual elements [56], sound separation [57], image segmentation [58], and different levels of supervision. These techniques may impose restrictions that could even make the translator produce more realistic results. Yet, such data structuring may lead the system to become in conflict with the objectives of the translation we propose, a situation which demands a commitment to diversity and imagination. Instead of relying on explicit human-generated knowledge, we explore the power of deep neural networks (DNN), including GANs, to model the aimed S2I translator. Given the task's complexity, designing and training these DNN models has proven challenging. Without minimizing the difficulties, except for constraining the number of sound classes used, we adopt a clear, exploratory approach, presenting a solution that gives clues on how to tackle the problem. Nonetheless, we also provide brief descriptions and suggest possible explanations for the observed phenomena. Furthermore, we opted to train the model without employing class-based losses. This choice aimed to prevent class-biased outcomes, allowing the translator to freely use the acquired multimodal knowledge without the potential restrictions imposed by a supervised approach. As discussed in the Introduction, the use of supervision for training the translator would restrict the aural and/or visual representations to pre-defined concepts, whether they are acquired by humans (for supervision) or extracted from unsupervised discrete representation learning (for self-supervision). A class-based loss function, for instance, may help the model to generate informative images. But it is hard to know if the model performed well due to a connection between modalities or if it is simply an image that fitted the class features. With our proposed translator, an informative image is necessarily generated through a successful connection between aural and visual modalities, because there is no clue at any stage of the process about the class of the input sound or the target image.