# EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance
## 1 Introduction  
With the skyrocketing popularity of a series of generative models, such as GPT4, LLaMA [1], StyleGAN [2], and Stable Diffusion [3], generative artificial intelligence has garnered significant attention in academia. Cross-modal generation, one of the most crucial tasks of generative AI, has received great attention for bridging the gap between different modalities. Image manipulation, a sub-task of cross-modal generation, has been extensively studied for its wide applications in digital social media. Image manipulation aims at editing images according to user intention. Recent work has innovated various kinds of methods for manipulating images through human language, such as semantic labels, text, or scene graphs [4]. Meanwhile, some researchers attempt to leverage audio to manipulate the generated images. An emerging line of work has sought to create multi-modal systems by leveraging contrastive learning mechanisms [5]. Among these methods, drawing inspiration from the text-based image manipulation method StyleCLIP [6], which leverages the representational capabilities of the contrastive language-image pretraining (CLIP) [7] model, [PERSON] et al. [8] extended this embedding space to encompass both audio-visual and text modalities. However, the methods of image manipulation based on audio information, particularly those utilizing music hybrid information, exhibit certain inherent drawbacks. A primary limitation of existing approaches is their focus on leveraging semantic elements derived from music, such as genre or rhythm, while largely neglecting the rich emotional landscapethat music, especially complex compositions, can portray. Music has the capacity to elicit a broad spectrum of emotions that extend beyond its semantic attributes. By overlooking these emotional nuances, the resultant images often do not capture the full depth and nuance of the emotions that the music aims to express. Consequently, the images manipulated in this manner struggle to align effectively with the music, leading to potential inconsistencies and instabilities in the image guidance effects. Furthermore, the guidance process in current methodologies typically relies on semantic information extracted from the music for singular-dimensional guidance, neglecting the application of multi-dimensional constraints. This oversight can result in generation outcomes that lack stability and fidelity to the multi-faceted nature of the music's emotional and semantic content. To overcome the identified challenges, we introduce a pioneering framework entitled the Emotion-aware StyleGAN Manipulator (EmoStyle). Initially, we deploy the Image-Bind [9] model to identify and retrieve images that are semantically aligned with the provided musical content, earmarking these images for further manipulation. Next, we extract the lyrical content from the music and conduct sentiment analysis to classify the emotional tone of the lyrics. Utilizing the insights gained from both the lyrics and their associated sentiment polarity, we craft an emotionally resonant instrumental music segment, intended to complement the primary musical content as auxiliary emotional music. In the final step, we employ pre-trained encoders to transform the musical and textual content alongside the retrieved images into a unified feature space. This convergence facilitates the execution of a hybrid latent vector alignment technique for precise and emotionally attuned image manipulation. In a nutshell, our contributions can be summarized as follows:  
* We introduce a sentiment-auxiliary semantic-guided image manipulation framework, termed the Emotion-aware StyleGAN Manipulator (EmoStyle), which efficiently leverages the affective information from music to manipulate images. * We devise a simple yet effective retrieval approach that selects images with minimal semantic gaps based on the similarity between music and images, which ensures the stability of subsequent image manipulation.