# MuIm: Analyzing Music-Image Correlations from an
## 1 Introduction  
Since the inception of human society, individuals have communicated complex ideas through a blend of semantic and non-semantic channels--the former providing rational clarity, while the latter delves into emotional depth, a duality artists have long harnessed for impactful expression [1; 2]. Today's technology empowers individuals to explore aristry through multimedia platforms, creating rich avenues for nuanced self-representation [3; 4]. Examining these diverse forms of multimodal data from both semantic and affective perspectives unveils the complexities of contemporary self-expression, highlighting challenges and opportunities within computer vision to capture this layered human experience [5; 6; 7; 8]. The prevailing body of literature for the audio and visual domain mainly concentrates on unimodal analyses, exploring their semantic [9; 10; 11; 12; 13; 14] and emotional dimensions [15; 16; 17; 18; 19; 20; 21]. However, their effectiveness is constrained by data quality, the inherent limitations of each modality, and ambiguous relations with other modalities. Consequently, there is an escalating academic focus on integrating multimodal [22; 23] data to enhance analytical performance and reconcile semantic and non-semantic interpretations of data. In audio-visual multimodal research, studies generally fall into two primary categories: semantic and non-semantic correlations. Semantic analyses are typically more straightforward due to their clear interpretability, while non-semantic studies encounter challenges related to subjectivity, limited training data, and a relative scarcity of multimodalresearch efforts. Recent work on semantic correlation [24; 25; 26; 27; 28; 29; 30; 31] has often employed video data or presumed links between modalities to perform contrastive learning with neural networks. Conversely, non-semantic correlation studies, which typically involve emotional labeling [15; 32; 33], frequently use valence-arousal (VA) metrics or basic emotion labels to train deep models on paired modalities. However, existing research has key limitations, primarily due to data constraints. First, the binary distinction between semantic and non-semantic correlations in audio-visual data limits a nuanced understanding of the complexities inherent in inter-modal relationships. Second, the use of overly simplistic categories or intricate valence-arousal models can obscure emotional clarity, especially since music or images within the same medium may convey contrasting emotions, making it difficult for both professionals and non-professionals to interpret artistic data accurately. Third, the scarcity of open-access datasets and the limited size of the available datasets pose significant challenges for training more effective and robust models. To address existing limitations, we recognize that music and visual data encompass multiple layers of information, as depicted in Figure 1. Current state-of-the-art methods predominantly focus on either the semantic or emotional dimensions of music-visual relationships, resulting in an incomplete analysis of their complex interactions. To bridge this gap, we introduce the Mulm dataset, which comprises two segments: one focusing on artistic music-visual content and another featuring realistic web-based music-visual content. To accurately capture the emotional nuances inherent in artistic content, we employ a hybrid approach that integrates both dimensional and categorical emotional models, offering a balance between comprehensive analysis and interpretability. Our dataset has been statistically validated and is available for further research, fostering advancements in the field. In summary, the contributions of this paper are threefold:  
* Exploring Multilayered Music-Image Correlation: To the best of our knowledge, we are pioneers in exploring the correlation of music-image data from an artistic perspective, incorporating multilayered information that encompasses both semantic and emotional dimensions within these modalities.