# MuIm: Analyzing Music-Image Correlations from an
## 4 MuIm Dataset
### Data Acquisition and Labeling
#### 4.1.2 Realistic Segment  
The realistic segment of the MuIm dataset encompasses web-sourced photorealistic images and diverse non-lyrical music, typically tagged with descriptive labels. This collection captures detailed semantic, affective, and contextual information, facilitating in-depth correlations between music and imagery.  
Data acquisition involved collecting 26,000 music clips from the internet, ranging from 8 seconds to 30 min in length, each tagged with content, mood, genre, theme, and movement labels. Additionally, 1.1 million user-tagged images were curated, containing labels with semantic, emotional, or contextual information. To ensure the consistency between the music and image data, only images with both sentimental/emotional and semantic tags were selected, resulting in a total of approximately 550,000 images. The data acquisition process for web-based multimedia data (music and images) involved several key stages:  
1. Web Media Metadata Collection: Due to the large volume of web data and limitations imposed by the hardware and bandwidth, URLs and human-defined tags were gathered to provide initial metadata, ensuring that only relevant data for our task were downloaded and processed.
2. Tag Pre-processing: Media URLs without tags were filtered out, and word tags were validated using transformer models, such as GPT-3, to ensure semantic accuracy and remove invalid tags.
3. Assurance of Emotional and Semantic Tags: Music data included separate mood tags, which, when combined with user-defined tags, provided accurate emotional information, while semantic tagging relied primarily on user-defined tags. In contrast, image data depended solely on user-defined tags for both emotional and semantic labeling. Due to potential noise, contextual variability, or inaccuracies in user-defined tags, a processing pipeline was implemented to refine and extract accurate semantic and emotional information for both image and music data. This pipeline, illustrated in Figure 4, consists of the following main components:  
* Multi-Layer Information: To obtain media with dual-layer information (emotional and semantic), sentiment analysis frameworks [56; 57] were used to select media that contained at least one sentimental tag in its user-defined tag list. This was particularly crucial for the image data, as they relied solely on user-defined tags.
* Information Check: To further enhance and validate the tags, media captioning models [13; 53] were employed to generate captions directly from the media (image or music).  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Aspect** & **Artistic Example (Image)** & **Musical Example (Music)** \\ \hline \hline \multirow{2}{*}{Theme} & Architecture & Baroque period \\  & & architecture-inspired themes \\ \hline \hline \multirow{2}{*}{Style} & Early Renaissance, Perspective & Baroque instrumental style \\ \hline \hline \multirow{2}{*}{Genre} & Art, Architecture & Classical Music, Instrumental \\ \hline \hline \multirow{2}{*}{Atmosphere} & Historical, Serene, Spiritual & Serene, Majestic, \\  & & Historical undertones \\ \hline \multirow{2}{*}{Instruments/Visual Elements} & Perspective lines, Linear & Pipe organ, Strings, Spacious \\  & perspective, Church interior & acoustic reverb \\ \hline \hline \end{tabular}
\end{table}
Table 3: Example of common semantics between artistic music and image.  
* Information Consistency: Finally, to ensure label consistency across the tags, a large-language model (LLM) [54] was used to determine the intersection between the human labels and AI-generated captions, ensuring consistency and accuracy. Examples of the image and music datasets for this process can be found in Tables 4 and 5.
* Data Post-processing: Based on the metadata processing, only relevant media (music/images) were downloaded. For the final pre-processing, a vision transformer (ViT) was employed to filter out duplicate images, and an NSFW detection model [58] was used to filter out NSFW content in the images.