Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings
Authors: Zhihuan Kuang
Shi Zong
Jianbing Zhang
Jiajun Chen
Hongfu Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-modal Learning
Music Description
Text Generation

Abstract: In this paper, we consider a novel research problem, music-to-text synaesthesia. Different from the classical music tagging problem that classifies a music recording into pre-defined categories, the music-to-text synaesthesia aims to generate descriptive texts from music recordings for further understanding. Although this is a new and interesting application to the machine learning community, to our best knowledge, the existing music-related datasets do not contain the semantic description on music recordings and cannot serve the music-to-text synaesthesia task. In light of this, we collect a new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions. Based on this, we build a computational model to generate sentences that can describe the content of the music recording. To tackle the highly non-discriminative classical music, we design a group topology-preservation loss in our computational model, which considers more samples as a group reference and preserves the relative topology among different samples. Extensive experimental results qualitatively and quantitatively demonstrate the effectiveness of our proposed model over five heuristics or pre-trained competitive methods and their variants on our collected dataset.

Under review as a conference paper at ICLR 2023
MUSIC-TO-TEXT SYNAESTHESIA: GENERATING DE- SCRIPTIVE TEXT FROM MUSIC RECORDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we consider novel research problem, music-to-text synaesthesia. Different from the classical music tagging problem that classifies music record- ing into pre-defined categories, the music-to-text synaesthesia aims generate descriptive texts from music recordings for further understanding. Although this is a new and interesting application the machine learning community, to our best knowledge, the existing music-related datasets do not contain the semantic descriptions on music recordings and cannot serve the music-to-text synaesthesia task. In light of this, we collect a new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions. Based on this, we build com- putational model to generate sentences that can describe the content of the music recording. To tackle the highly non-discriminative classical music, we design a group topology appoogyypreservation preservation loss in computational model, which considers more samples as a group reference and preserves the relative topology among different samples. Extensive experimental results qualitatively and quantitatively demonstrate the effectiveness of our proposed model over five heuristics or pre- trained competitive methods and their variants on our collected dataset
INTRODUCTION
Multi-modal learning has drawn great attention in recent years and has been very developed in diverse applications, since our physical world is naturally composed of various modalities. Visual frames in videos are matched with text captions and these pairs have been widely used for video- language pre-training (Sun et al., 2019] Luo al., 2021] Li et al. 2020): Kinects employ the RGB camera and the depth sensor for action recognition and human pose estimation (Shotton al.. 2011
Carreira & Zisserman] 2017); autonomous driving cars integrate the visible and invisible lights by the camera, radar and lidar for series of driving-related tasks (Buehler al., 2007 Torres al.) 2019): cross-modal retrieval aims to understand and match text with the existing textual repository and other modalities meet users' queries (Nagrani al. 2018 (uuii et al. 2018 Zeng al. 2021 language grounding learns the meaning of natural language meaning by leveraging the sensory data such as video or images (Bisk et al., 20200 Thomason al. (021).
Besides the above studies that employ multi-modal data to jointly achieve the learning task, trans- lating information among different modalities, also known as synaesthesia, is another crucial task in the multi-modal community, where the text with its good compatibility and presentation ability, has become the intermediary modality interaction. Various methods for synaesthesia between text and other modalities have been studied. Speech recognition can be directly regarded translation between the text and audio modality (Shen et al.] 2018). Image captioning extracts the high-level visual cues and translates them into descriptive sentence to describe the image content, while some studies consider the inverse process of image captioning by converting a semantic text into the vi- sual image (Huang al. 2021 Xu et al. 2015). Different from the existing modality translation studies, in this paper, we consider a novel problem, the music-to-text synaesthesia, i.e., generating descriptive texts from music recordings.
Recently, there are some pioneering attempts that build the connections between music recordings and tags at the initial stage. Cai et al. (2020) formulate music auto-tagging as captioning task and
Our code and data resources are available Musi wusicceextttttaaeec Musiiccleettyynaetthesia

---

Under review as a conference paper at ICLR 2023
automatically outputs sequence of tags given clip of music. Zhang et al. (2020) use keywords of music key, meter and style to generate music descriptions, which can be used for caption gen- eration. However, we argue that descriptive texts contain much richer information than tags, thus providing a hetter understanding of music recording. Moreover, we notice that tags might have a biased interpretation. Figure | presents
music recordings with the same music tags, but the opposite sentiment orientation of the text. The first one expresses positive sentiment by describing the music as "peaceful" and "beautiful," while the second one uses tokens including "sadness" and "loss" to express a negative sentiment. It is clear that music tags are insufficient for describing the content of a music piece.
Contributions. In this paper, we propose new task of generating descriptive text from music recordings. Specifically, given a music recording, we aim build computational models that can generate sentences that can describe the content of the music recording, as well as the music's inherent sentiment. The major contributions are summarized as follows:
From the research problem perspective, different from the music tagging problem, propose the music-to-text synaesthesia, a cross-modality translation task that aims at converting a given music piece to a text description. To our best knowledge, the music-to-text synaesthesia is novel research problem in the modti-modal learning community.
From the dataset perspective, the existing music-related datasets do not contain the semantic description of music recordings. To build computational models for this task, we collect new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions.
From the technical perspective, we design group tional model to tackle the non-discriminative music representation, which considers more data points as a group reference and preserves the relative topology among different nodes. Thus can better align the music representations with the structure in text space.
preservation loss in our computa-
From the empirical evaluation, extensive experimental results demonstrate the effectiveness of our proposed model over five heuristics or pre-trained competitive methods and their variants on our collected dataset. We also provide several case studies for comparisons and elaborate the explorations on our group
loss and some parameter analyses.
2 RELATED WORK
Multi-modality Learning. The goal of multi-modal machine learning is to build computational models that are able to process and relate information from different modalities, such as audio, text, and image. [Baltrusaitis et al. (2019) describes five challenges for multi-modal learning, namely: learning representations, translating, aligning, fusing, and co-learning from/between dif- ferent modalities. A large portion of prior works has focused on modality fusion, which aims at making predictions by joining information from two more modalities. Applications include audio-visual speech recognition (Afouras et al., 2018), visual question answering (Goyal et al. 2017]. emotion recognition, and media summarization. We now briefly review the literature and refer readers to Guo et al. (2019); Baltrusaitis et al. (2019) for more complete surveys.
Translation between Modalities. Beyond the applications of multi-modality translation we men- tion in the introduction section, here we introduce three common frameworks of multi-modality translation. (1) Encoder-decoder models directly learn intermediate representations used for pro- jecting one modality into another. Zhang et al. (2017) adopt sketch-refinement process to generate photo-realistic images for text-to-image synthesis tasks. Wang et al. (2021) designed framework for end-to-end dense video captioning with parallel decoding. (2) Models with joint representations fuse mudti- features by mapping representations of different modalities together into shared semantic subspace. [Sun et al. (2019) proposed ViLBERT, which extends BERT architecture to multi-modal two-stream model, that learns task-agnostic joint representations of image content and natural language. |Habibian et al.] (2017) designed an embedding between video features and term vectors to learn the entire representation from freely available web videos and their descriptions. (3) Representations in coordinated representations based models exist in separated spaces, but are coordinated through a similarity function (e.g.. Euclidean distance) or a structure constraint. These works include Wang et al. (2017). which presents method to learn common subspace based on adversarial learning for adversarial cross-modi cross-modal retrieval. [eng et al. (2018) proposes modality- specific cross-modal similarity measurement approach for tasks including cross-mode moddll retrieval. In
2

---

Under review as a conference paper at ICLR 2023
Movement Andante espressivo (by Charle Stanford)
Tags: - minor, instrumenc string, ensemble - quartet, tempo mediu
Text: Parhaps the work's center of gravity is the slow movement, Andante espressivo. The motif given peaceful, and extraordinanll beautiful frestment. The stormy midcle section makes a very strong impression.
Movement Tempo moderato (by Wilhelm Stenhammar)
Tagsi mode minor, instrument string onsemble - quartes, campo medium
Text: movement to Quartet No. Tempo moderato un poco rubato, though not funereal, clearly conweys the feelings sedness and loss. The tempo never speeds and the players are warned off keoping strict tempo.
Figure/ 1: Samples of classical music and corresponding text descriptions in our collected dataset. The first piece is from String Quartet 2o.2 in A minor Op.45 composed by Charles Stanford, and the second is from String Quartet No.6 in D minor Op.35 by Wilhelm Stenhammar. While two samples have exactly the same music tags, clear that the two text descriptions have different sentiment.
this work, we experiment with different losses on the coordinate model, as it achieves the best per- formance among all three different types of models.
Music Related Datasets and Tasks. The existing music-related datasets mainly contain simple music tags. AudioSer dataset (Gemmeke et al., (017) is large-scale collection of numan-labeled 10-second sound clips (not music recording) drawn from YouTube videos. This dataset only has descriptions for categories, not for individual sounds. The MTG-Jamendo dataset (Bogdanov al. 2019) contains over 55,000 full audio tracks with 195 tags ranging from genre, instrument, and mood/theme categories, Oramas et al. (2016) describes a dataset containing reviewers from Amazon for albums. However, users' reviews may not necessarily describe the actual contents of music recordings. Cai et al. (2020) formulated the music tagging problem as a multi-class classification problem. A dataset called MajorMiner is used, with each music recording associated with tags collected from different users. Zhang et al. (2020) studied bidirectional music-sentence retrieval and generation tasks. The used dataset contains 16,257 folk songs paired with metadata information, including select key, meter and style as keywords. Text describing music recordings has limited style and mainly focuses only some specific information.
We notice the studies on music cuptioning-relater tasks address a similar research question (Choi let al., [2016] Gao et al. 2022 Manco al. (022).. Specifically, Manco et al. (2021) used a private production music dataset, with their music clips of length between 30 and 360 seconds and cap- tions contain between 3 and 22 tokens. Manco et al. (202I) also used an encoder-dccoder network consisting of multimodal CNN-LSTM encoder with temporal attention and an LSTM decoder.
3 DATA COLLECTION AND ANALYSIS
Since we address a new problem that generates descriptive texts from music recordings in this paper, dataset containing aligned music-text pairs is required for model training. Although there several public music/audio datasets with tags or user reviews (See Section 2). unfortunately, they do not meet our research problem for the following reasons: (1) From the text side, current datasets only have pre-defined tags for music pieces, rather than descriptive texts for music contents. (2) From the audio side, some clips are too short without a musical melody. In light of this, we collect a new dataset for the music-to-text synaesthesia.
Data Collection and Post-Processing. We collected the data from EARSENSE a website that hosts database for chamber music, where Figure I shows an illustrative example of the music-text pairs. A typical music composition in chamber music contains several movements. Each movement has title that normally contains tempo markings or terms such as minuet and trio; in some cases, has a unique name speaking to the larger story of the entire work. As movements have their own form, key, and mood, and often contain complete resolution or ending, we will treat each movement as the basic unit in this work. For example, Ludwig van Becthoven's 's Piano Sonata No. in C minor, Op. 13 (also referred as Sonata Pathétique) contains three movements: (I) Grave (slowly, with solemnity), (II) Adagio cantabile (slowly, in a singing style), and (III) Rondo: Allegro (quickly). EARSENSE also provides comprehensive meta information for each music composition, including
http://earsense.org/
3

---

Under review as a conference paper at ICLR 2023
composers, works and related multi-media resources. There is also associated introductory article from professional experts, with detailed explanations, comments or analyses for movements.
By dividing the music compositions into movements according to their meta information, col- lected 2,380 text descriptions in total, where 1,955 descriptions have corresponding music pieces. Each movement has its specific title, such in-domain words indicating the types of music. This kind of information is useful when constructing a model as classical music follows some fixed pat- when composed. In general, tempo markings are specific but may differ from countries. In this work, we refer to Wikipedia and convert Italic, French, German and other tempo markings into universal four categories from slow to super fast. These categories are then added for movements as tags, by directly checking whether it contains tokens in our list.
Preliminary Exploration. The lengths of the 95% collected music pieces vary from 2.5min to 14min, which correspond to the descriptive texts with 14 to 192 tokens. The longest music length up to 37 mins. We provide more details of these statistics in the Appendix.
500 1000 1500 1865
500
1000
1500 1955
0.06
500
0.98
0.04
1000
1030
0.96
0.
1600
1600
Beyond the above basic statistics, we further ex- plore the pairwise similarity and check whether similar music pieces share similar textual de- scriptions. Figure 2 shows the pairwise similar- ity matrix of music and text, respectively, where the cosine similarity on the music rep- resentations derived from a self-reconstruaciinn auto-encoder and use BLEU score to calculate the raw text similarity. We notice that the texts are highly discriminative, where 96% of the pairwise similarities are below 0.06. However, all music representations have over 0.95 similarity due to the high similarities among classical music. It inevitably increases the difficulty of learning a map- ping function for the music-to-text synaesthesia task that the similar music pieces match divergent texts. Thus, a natural idea is to employ the text to guide the learnt music representations to make them more discriminative, so the cross-modale moos-moodllity transformation would be easier. The research ques- tion we explore in this paper is: How can we employ the music-text pairs to learn the informative representation for music pieces and achieve the music-to-ted synaesthesia?
(a) music similarity
(b) text similarity
Figure 2: representation by a self-reconstruction autoen- coder and raw text by cosine and BLEU score.
Pairwise similarity matrices of music
4 MODEL
The music-to-text synaesthesia is a cross mrosaitodality translation problem. Given n paired tuple (m;, 4) with 1 < n, where m; denotes the tuple i's music and 4 for its corresponding text, music-to-text synaesthesia model M + T builds mapping from the music to the text space, where M and T present the music and text space, respectively. In this section, we first pro- vide a general overview of our model structure, followed by our proposed group
Music
Text
Decoder
Decoder
GTP Loss
preservation loss tackle the non- discriminative music representation.
Music
Text Encodeer
Cross-Modality Translation Model. Figure shows our translation model, which is mainly based on the coordinate model (Peng et al.) 2018) and our proposed group topology- preservation loss to achieve the translation between two modali- ties by learning the music/text latent space and their mapping by using three independent auto-encoders.
Music
Figure 3: Framework for our
music-to-text synaesthesia task.
Learning Music Representation. We choose a convolutional neural network structure as our music feature extractor. Specifically, use the spectrum of the raw music as inputs, where CNN encoder and a waasposed-CNN decoder are connected for the music reconstruction task. Let f()) as the music encoder and f'(-) the music decoder. We have the loss function for music representation
htipa: httpa://en.wii wikk .rgg/wt kt /Tmppo
4

---

Under review as a conference paper at ICLR 2023
Text Space
Negatvi Foi
Posttiv Pair
Group Topology Preservation
Guidance
Positve Par
Music Space
Heg Po
6
(a) Original
(b) Triplet
(c) GTP (Ours)
Figure 4: Our group copology-preservation (GTP) loss. (a) The structure of data points in the original text and music spaces. (b) The triplet loss is able to enforce the similarity between pairs of points, but fails to maintain the relative positions with other data points. (c) Our proposed GTP loss considers group reference for each point and aligns the music representations with their structure in text space. learning as follows:
mm2m
||m, J'(((m.)))l..
Learning Text Representation. Similarly, in the text field, transformer maaseddaeer-beddd encoder and decoder are combined to train text reconstruction model. Let g(-) be the text encoder and g g'(-) be the text decoder. We have the loss function for text representation learning as follows:
E12 CrossEntropy(t,, o' (g(())
Transforming From Music to Text. With the above music and text representation, an extra auto- encoder is employed to achieve the mapping M from music to text.
Em = [lf(mm) M(g(ts))ll
(3)
(GTP) Loss. Although the above coordinate model can achieve the music-to-text transformation, we notice that the music representations by the auto-encoder suffer from high non-discrimination, which further prevents the effective transformation. In light of this, we design a novel Group Aapology-Preservatiin (GTP) loss to learn the discriminative music repre- sentation. Due the relatively high discrimination in the text space, we aim employ the text guide the music representation generation.
Group
To better illustrate our point, we provide a visualization of our designed group oopology-preservatiie loss and the comparison with the triplet loss in Figure where (a) shows the original representation space of music and text derived from the auto-encoders, (b) shows the effect of the triplet loss and (c) shows the effect of our GTP loss. When applying the triplet loss to node #3 in Figure {b), we observe that nodes #3 and #4 get close and nodes #3 and #5 are more separated. However, its relative positions as opposed to other data points are still different from the structure in the original text space. Figure Me), our proposed GTP loss considers more data points as a group reference and preserves the relative topology among different nodes, thus can better align the music representations with the structure in text space.
Formally, given a paired tuple (m,, 4.) and batch B = {m, 4) containing other k paired tuples as the group reference, our GTP loss can be defined as follows:
(4) where cos is the cosine similarity and BLEU score for the musie and text's similarity calculation and CON is a concatenation function that flats the input elements and returns a row vector, and softmax normalizer across all elements. Our GTP enforces the given tuple and its group reference share the similar topology structure in both the hidden music space and the original text space, further separating the highly similar music representations. Moreover, the GTP loss not only captures the relationship between the given tuple and other samples in the group reference but also considers the relationship within the group reference.
{GTP softmax Coottmax(CON(coode (m,), ( (m,))) softmax (CON(BLEU((
Overall Objective Function. Our overall objective function can be written as follows:
(Emzm + leze Emzt allur) (((()))
(5)
5

---

Under review as a conference paper at ICLR 2023
where θ is the whole learning parameters including f. f 9. g' and M, Ω the parameter regular- izer and a, 3 are trade-off hyper parameters among different losses in the objective function.
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Here we propose five competitive methods and their variants for the music- to-text synaesthesia, which are in two main categories, heuristics and pre-trained methods.
Comparison Models.
Heuristics methods. In our collected dataset, each music recording is associated with 4 types of tags, including mode, instrument, tempo and ensemble (see Table A2 in the Appendix). We note these tags normally contain information that categorizes the characteristics of the recordings. Motivated by this observation, we design two simple baselines that are based on heuristics, TAGS (kNN): Inspired by the & nearest neighbor clustering, TAGS (KNN) assigns the text from the closest tags for each piece of music. Specifically, we directly use the text description from music recording which has the largest number of same tags with our given music. TAGS (REPRESENTATIVE): Alternatively, we first divide all music pieces into separated groups based four categories of tags. For each tag combination, we select representative sentence, which has the highest BLEU score within its equivalent classes. These representative sentences can be formed as a text description for given music with the same tag combination.
Pre-trained methods. ENCODER DECODER. The Encoder-Decoder model consists of a music CNN encoder and a text transformer decoder. The CNN encoder takes the spectrum of music as input and its output is then directly sent to the text transformer decoder to generate text de- scriptions. JOINT. The Joint Model is based on the assumption that information from different modalities is the same. A music encoder and text encoder and text inputs. Then the encoded outputs are sent to an adversarial discriminator to identify the encoding source, forcing the alignment of representation from different modalities. Finally, the encoding vector from both music or text can be decoded into text description without modal- ity source information. COORDINATE. Coordinate model representing different modalities in different spaces. To do so, two autoencoders are used for music and text representation. And the text decoder also required learn a mapping of music to the description text. Moreover, based on COORDINATE, we propose its variants with additional losses for better music represen- tation learning. Contrastive loss(Chen et al. 2020 builds positive samples by pitch shift, time stretch, randomly mask and noise addition Won al. 2020) to increase representation robust- ness. Pairwise loss pulls a pair of music representations together with similar texts. And triplet (SssrSchroff et al., (015)) is an improved version that set margin to pull similar samples and push other negative samples away. Figure A3 in the Appendix shows the differences among the Encoder Decoder, Joint Model and Coordinate Model.
used to first encode the music
Implementation. From the music side, the original classical music is first converted into spectrogram which has fre-
Original
(())) (roqunnc((Ht))
1500
quency axis of 256 dimensions and a flexible temporal axis. It is then encoded into T 768 by CNN encoder, where T is the music length in seconds. Both the music encoder and decoder contain layers. From the text side, the text de- scriptions are tokenized by BERT tokenizer and encoded into
3000
4500
6000
Reconstrution
1500
L x 768, where is the number of tokens. The text en- coder and decoder consist of 5 layers. As the input music and text are of varying length, both music and text encodings are divided into 20 segments evenly, and mean pooled into 20 x 768 vectors as a fixed-length representation. The same number of layers for music and text encoders are used for Encoder-Decoder and Joint Model baselines.
3000
4600
6000
Gmm(s)
Figure 5: Spectrum diagrams from the original music (top) and recon- struction (bottom) of our pre-trained music autoencoder.
For our model, we first pre-train music and text auto-encoders on our constructed dataset. To pre- vent data leakage, the text auto-encoder is pre- pre-trained without the test data. Moreover, these music and text auto-encoders capture good representation in the reconstruction task. Figure shows the spectrum diagrams of the original and reconstruction music from music representation. which con- tains the main information of original music. Meanwhile, the text-autoencoder reconstructs from

---

Under review as a conference paper at ICLR 2023
representation with average BLEU score of 79.66. During training, set the learning rate to be 5e and the batch size is 8. implemented by gradient accumulation. The hyperparameters in our model are set to be a 500, 3 = and Is 32. Considering the computational budget, our l grouped sample representations are implemented by a queue according to MOCO (He et al. 2020), where employ momentum encoder with momentum 0.999.
Evaluation. As the size of our dataset is rather small, we evaluate the model performance using the 5-fold rross-validation for all our experiments. We use the BLEU score (Papineni et al. (002) as our main evaluation metric to compare our generated captions with the reference text descriptions. BLEU score has been confirmed to have high correlation with human judgements, and it is widely used in natural language generation tasks, such as machine translation and text summarization
5.2
MAIN RESULTS
For each data split, we run 5 experimental trails and then report the average and standard deviation of the BLEU scores in [ Note that the ground text descriptions for classical music are sometimes hard to digest even for humans (see the cases in Table 2 it is thus not totally surprising that the performance of our model is generally low
We observe that heuristics baseline methods have lower performance compared to learning-based models. is intuitive as heuristics-based methods are not able generate text descriptions that are adjusted to given music contents. Among all three representative models for multi-modal tasks, COORDINATE model achieves the best performance with an average BLEU score of 6.42. When further adding different losses to the training objective of the coordinate model, we observe similar performance for using triplet loss and even drops for contrastive loss and pairwise loss; while there is a significant performance improvement when using our proposed GTP loss, achieving the highest BLEU score of 6.76. Our experimental results compared to baseline methods are statistical signifi- cant with p-value < 0.001. Some music recordings on earsense have second comment by another commentator. We collect these data to be another dataset with 50 records and validate our models on it. The BLEU score of the baseline Coordinate model is 8.54 and our GTP loss model is 8.81.
To better interpret this experimental result, in Figure 6 we plot the music and text similarities of a pair of different music-text tuples along with the histograms the music similarity when applying different losses. We observe that the output representations from pre-trained music autoencoder have high similarity, with most above 0.90 in Figure G[). a). It is because the high frequency of all music is almost blank while classical music also has a similar low frequency with several limited instruments. When using pair loss, we observe the music representations simply gather together and become indistinguishable (shown in Figure b)). Meanwhile, using our proposed GTP loss helps maintain a structure among music representations. As shown in Figure c). the music similarities vary from -0.05 to 0.4 and the range is much wider than that of other models. Also, the representations spread out and become more distinguishable, which naturally helps the performance of text generation. validates the motivation of our designed GTP loss that when considering more data points as group reference, the model is able to capture the relative topology among different nodes, thus can better accomplish the transformation from different modalities.
We further present several examples in Table 2] showing the comparisons between ground truth music descriptions and the generated ones from COORDINATE with without our proposed GTP loss. We observe that COORDINATE and OURS are able to identify basic characteristics of the input musics, while our model in general could capture more subtle differences. For example, in Case #1, both models correctly identify the tempo category "allegro," while our model further captures the music instrument of "cello" and genre of "sonate." " In Case #2, our model correctly captures two themes in one movement, where the "bright and lively" first theme is recognized as "march like scherzo." and the "dreamy" second theme is described be "romantic".
It can be observed that, for objective facts, models can simply and correctly tell the correct term, while our model can further capture more details. When comes to subjective description, our model could capture the sentiment orientation of music themes, although it is still a challenging task
*The typical BLEU score for machine translation task from 10 to 40 Vaswani et al. 2017).
'we do not report the performance of MusCaps (Manco et al. 2021 due to their low performance. We conjecture it comes from the different music feature backbone.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Comparisons of different models for music-to-text generation task using BLEU metric. For all neural-bace basad methods, we run experiments 5 times for each split and then report average scores and standard deviations. in triplet loss refers to different pair numbers used.
Split #1
Split #2
Split #3
Split #4
Split #5
Average
TAGS (&NN)
3.95 3.75
3.73 3.65
3.6 4.06
3.84 3.51
3.57
3.74
TAGS (REPRESENT.)
3.13
3.62
ENCODER DECODER
5.86
6,3 1.31=0.2m #0.27
6.36
5.77
6.59
6.18
JOINT
5.84
6.3 20.24
6,48
5.83
6,27
6.15
COORDINATE
6.41
6.70
6.20
6. 6.1020.11
6.6610.24
6.42
COOR. Contrastive loss
5.71 =0.36
2.80
6. 6.74
4.11
4..84
6.07
COOR. + Pairwise loss
6.20
6.76
6.53
5.97
6.53
6,40
COOR. + Triplet loss (k=1)
2.10
6.47 10.13
6.74
6,07 20.13
6,95
6,56
COOR. + Triplet loss (A=32)
6.36
6,59
6,50
2.94
4.71
6.42
OURS
6.94
20.15
6.91 20.25
6.37 20.24
10.1
6.76
0.99
0.99
0.4
music similarity
0.97
0.97
0.98
0.98
0.2
0.1
0.84
0.94
0.93
0.93
0.0
0.92
0.92
-0.1
0.91
0.91
0.00 0.05 0.15
600 0.00 0.05 0.10 0.15 text similarty
0.05 0.10 0.15
300
similaritt
num
num
text similatty
num
(a) Pretrained music autoencoder
(b) COOR. Pairwise loss
(c) OURS
Figure 6: Music and text similarities a pair of different music-text tuples and histograms on the music similarity, where the music similarity is caleulated by the cosine similarity on the learned representation and the text similarity is calculated by the BLEU score on the original texts. (a) Pretrained music autoencoder; (b) COORDINATE with pairwise loss; and triplet loss has similar histograms; (c) OURS model (with GTP loss).
due to no sentiment supervision and expression diversity (see Section 5.3 for further analyses of the sentiment transfer ability between different modalities). We also notice some grammar mistakes and fixed pattems appear in generated sentences. seems that even with sufficient pre-training it is still hard to learn professional expression of both correctness and fluency from limited dataset.
5.3 IN-DEPTII ANALYSES
Effects of Hyperparameters. We are interested in the effect of the number of samples considered in our GTP loss. To do this, vary the number of k from 4 to 2,048 and plot the BLEU scores in Figure 7(a) We observe that the performance of our model first increases and achieves its peak when k is chosen be 32; the performance then drops when is larger. It provides empirical evidence that considering more information from other data points do help improve the model performance. However, a larger is not always better for which will make older samples in the momentum encoder queue more outdated. We observe a similar pattern when varying the scale of the hyperparameter from 1e-2 le5 in Figure 7(b) The model yields the best performance when a = 500.
Transfer A Ability of Sentiment. Since our task is music-to- text synaesthesia, we care about whether our model could learn a transfer capability from the music domain the text domain. To explore this, we analyze whether the generated text descriptions are able to reflect the sentiment from the original ground truth text. We select popular sentiment classifie that could detect variety of 7 sentiment categories and then apply this classifier to identify the sentiment in reference text and text descriptions generated by our model. Figure (() shows the sentiment distribution of our generated texts and the original reference texts. We observe skewed distribution that majority of
httpat//hugggret waw..a//................. co/1- www.tttaaaannntto leoo on-engi 1ih-df st robeetta-aase
8

---

Under review as a conference paper at ICLR 2023
Generated text descriptions between different models given a classical music piece, fol- lowed by their sentiments [joy/neurtal/sadness) in the end. We use different colors to highlight phrases: blue for noun phrases that is objective facts and yellow for subjective adjective phrases.
Table 2:
Ref.: rousing allegro vivace finale begins with a sweetly singing adagio introduction that very briefty seems to recall extended introductions the earlier op. 5 sonatas and provides only hint of slow movernent that will otherwise have wail for the last cello sonata for its fully independent flowering. [joy] COORDINATE: the movement, allegro moderato, begins with fugue with fugue. [neutral]
Case
OURS: the movement, allegro moderato. begins with theme of the main theme of the main theme in the cello. the main theme is kind of the middle section, the work of the movement. the music is a fugue with bit more lyrical sccond theme to the sonata form pizzicato. [joy]
Ref:: the movement, allegro molto con brio, is based on two subjects, the first is bright and lively while the development is ingenious and an exciting coda caps this
second is dreamy with an improvisational first rate work. [joy]
Case COORDINATE: the movement, allegro moderato, begins with theme in the main theme. [joy]
OURS: the movement, allegro molto. is series of variations that begins with a march like is a scherzo and sense of the main theme. in the movement. it is very romantic second theme. sense of the movement. [juy]
Ref:: the movement, allegro moderato, begins with somewhat passionate, very characteristic quartet like melody which leads to an ingratiating, lyrical second theme. [joy]
Case #3
COORDINATE: the movement, allegro. begins with lengthy introduction. [ncutral]
OURS: the movement, allegro. begins with lively allegro brimming with long series of the main themee. begins a very romantic melody, music movement, [jay]
7.0
Reference
7.0
1000
modal
8.8-
800
6.6
6.6
mum
800
6.4
400
6.2
6.0
200
8 18 64 128
512 20242048
0.1
10
100
500
184
1e5
neu.
sad
Tea.
ang. sur.
dis.
(a) Number of samples ls GTP
(h) Parameter sensitivity
(c) Sentiment distribution
Figure 7: (a) BLEU score when varying the number of neighbors k. (b) Parameter sensitivity on a.
(c) Sentiment distribution of our generated texts and the original reference texts.
descriptions from both references and generated models are classified as "joy," while the left others tagged as having sentiments like "neutral," "sadness," and "fear." These experimental results consistent with our intuitions that classical music normally brings people with happiness and joy. We observe that the distributions for the sentiment labels from reference and our model are similar, with Pearson correlation coefficient up to 0.96.
6 CONCLUSION
In this paper, we proposed a novel task of music-to-text synaesthesia, the goal of which is to gener- text descriptions for a given music piece. As current existing datasets normally do not contain semantic descriptions for music pieces, we collected new dataset that contains 1,955 classical mu- sic recordings with professional text descriptions. We also proposed a group applogyypreeervatiin loss that preserves relative group topology among music and text, given the fact that classical mu- sic is non-discriminative. We conducted experiments on five heuristics or pre-trained competitive baseline methods, based on existing multi-modal literature. Compared with other sample level con- straint losses, our model with GTP loss extracted better music representation and generated better text descriptions. We further performed extensive experiments including case study and sentiment analysis to demonstrate the effectiveness of our proposed music-to-text synaesthesia model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine telligence, 2018.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Moreney. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, (2): 423-443, 2019.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Proceedings of Conference on Empirical Methods in Natural Language Processing, 2020.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg- ntg-jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, Inter- national Conference on Machine Learning, 2019.
Martin Buchler, Karl lagnemma, and Sanjiv Singh. The 2005 DARPA Grand Challenge: The Great Robot Race. Springer Publishing Company, Incorporated, 1st edition, 2007.
Tian Cai. Michael Mandel, and Di Hc. Music autotagging as captioning. In Proceedings of Workshop NLP for Music and Audio, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recogni- tion, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning. 2020.
Keunwoo Choi, Gy"orgy Fazekas, and Mark B. Sandler. Towards music captioning: Generating music playlist descriptions. ArXiv, abs/ 608.6080 04868, 2016.
Wenhao Gao, Xiaobing Li, Cong Jin, and Yun Tie. Music question answering:cognize and perceive music. In 2022 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1-6, 2022. doi: 10.1 00091CCMEW664482222299995555
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore. Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeles dataset for audio events. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2017.
Yash Goyal, Tejas Khot, Douglas Summers-S Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey, IEEE Access. 7:63373-6333 2019.
Amirhossein Habibian, Thomas Mensink, and Cees G. M. Snoek. Video2vec embeddings recognize events when examples are scarce. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 39(10):089-2103, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi- directional image and text generation. In Proceedings of ACM International Conference on Mul- fimedia, 2021.
10

---

Under review as a conference paper at ICLR 2023
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for Video+Language repeeseprrsnnntiioon pre-training. In Proceedings of Conference on Empirical Methods in Natural Language Processing, 2020.
Jianjie Luo, Ychao Li, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mci. Coco-bert: Improving video-language pre-training with contrastive cross-modal matching and denoising. Proceedings of ACM International Conference Multimedia, 2021.
Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas. Muscaps: Generating cap- tions for music audio. In 2021 International Joint Conference on Neural Networks (IJCNN), pp- 1-8, 2021. doi: 10.1 10.119//CNNN5228..20100999999 20.3346.....
Ilaria Manco, Emmanouil Benetos, Elio Quinton, and Gy"orgy Fazekas. Contrastive audio-language learning for music. ArXiv, abs/2208. 12208, 2022.
A. Nagrani, Albanie, and A. Zisserman. Seeing voices and hearing faces: Cross-modal bio- metric matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
Sergio Oramas, L. Espinosa-Anke, A. Lawlor, Xavier Serra, and H. Saggion. Exploring customer reviews for music genre classification and evolutionary studies. In Proceedings of International Society for Music Information Retrieval Conference, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of Annual Meeting of the Association for Com- putational Linguistics, 2002.
Yuxin Peng, Jinwei Qi, and Yuxin Yuan. Modality-seecific cross-modal similarity measurement with recurrent attention network. IEEE Transactions on Image Processing, 27(11)55555-5599 2018.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: unified embedding for face recognition and clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, R. J. Skerry-Ryan, Rif A. Saurous, Yannis Agiomyr- giannakis, and Yonghui Wu. Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2018
Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, Moore, A. Kipman, and Blake. Real-time human pose recognition in parts from single depth images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2011.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: joint model for video and language representation learning. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, 2019.
Didac Suris, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giro-i Nieto. Cross-modal embeddings for video and audio retrieval. In Proceedings of the European Conference on Com- puter Vision Workshops, 2018.
Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer. Language grounding with 3d objects. In Proceedings of Conference on Robot Learning. 2021.
Lucas Tabelini Torres, Thiago M. Paixão, Rodrigo F. Berriel, Alberto F. De Souza, Claudine Badue, Nicu Sebe, and Thiago Oliveira-Santos. Effortless deep training for traffic sign detection using templates and arbitrary natural images. In International Joint Conference on Neural Networks, 2019.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL http:::tt//s...........t httttss://proceeddine www.nnnw......iieeeeeeeeee wwt.p/pper/21///iieee.
pdf
Bokun Wang. Yang Yang. Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross-modal retrieval. Proceedings of ACM International Conference on Multimedia, 2017.
Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. Enn-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Minz Won, Andres Ferraro, Dmitry Bogdanov, and Xavier Serra. Evaluation of cnn-based automatic music tagging models. arXiv preprini arXiv:2006.0073: 000550000..7, 2020.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of International Conference on Machine Learning. 2015.
Yawen Zeng, Da Cao, Xiaochi Wei, Meng Liu, Zhou Zhao, and Zheng Qin. Multi-modal relational graph for cross-modal video moment retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adver- sarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2017.
Yixiao Zhang, Ziyu Wang, Dingsu Wang, and Gus Xia, BUTTER: A representation learning frame- work for bi-directional musie-sentence retrieval and generation. In Proceedings of Workshop NLP for Music and Audio, 2020.
12

---

Under review as a conference paper at ICLR 2023
A DATA ANALYSIS
We summarize the statistics of our newly collected dataset in Tables A1 and A2 Figures A1 and A2 There several advantages and unique features for this dataset: (1) Music pieces semantically hard enough, which are provide space for generating text with rich information. (2) The descriptive texts are generated by music professionals, thus freestyle and diverse.
Table Al: Statistics of our collected dataset
Table A2: Four categories of tags in our dataset
of unique music compositions of unique composers
786
Tag
Type
304
mode instrument tempo ensemble
major. minor
of music-text pairs
2,380
string, wind, piano
avg. of tokens per text description
45.7 1: 41.2 305
slow, medium, fast, super faster sonate, trio, quartet, quintet or more
avg. of each music piece (seconds)
600
count
count
100
200
500
1000
100
200
(a) Music Length
(b) Token Numbers
Figure A1: Distribution of music length and token nums
1000
800
count
600
400
200
None
major
minar
None
string
piano
wind
Nome
sonata
tro
quarter
above
None
slow
meriun
fact
faster
(a) mode
(b) instrument
(c) ensemble
(d) tempo
Figure A2: Distribution of four tag categories
B STRUCTURES OF BASELINE MODELS
We visualize the structure of all baseline models in Figure A3
Text
from music or text?
Text
Music
Text
TextDecoder
Discriminator
TextDecoder
MusicDecoder
TextDecoder
MusicEncoder
MusicEncoder
TextEncoder
MusicEncoder
TextE
1 Music
Music
Text
Music
Text
(a) EncoderDecoder
(b) ointModel
(c) CoordiinaeMModel
Figure A3: Structures of baseline models.
13


Reviews:
[{"review_id": "rEyOvwi7jD", "reviewer": "Reviewer_fsng", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a music captioning system that was trained on a classical music dataset. A modification on loss is proposed to improve the performance. ", "summary_of_the_review": "Title - Is there any reason to avoid music captioning?\nIntroduction - Similarly, the recent progress in music captioning (and even audio captioning maybe) is missing.\nSection 2 - I appreciate the preliminary exploration. \nSection 3 - Good approach to see the problem a cross-modality translation problem. The proposed system and GTP loss seems sensible to me.\nOn GTP section \n  - \"by the autoencoder suffer from high non-discrimination.\": Can you elaborate this?\n  - \"Due to the relatively high discrimination in.. ..generation.\": I get it, but it'd be nicer if this is elaborated, too.\n- Is \"node\" a music item?\nSection 4\n - Choice of BLEU - It makes sense. But it would be even better if there's some example about how BLEU works for this problem / dataset.\nSection 4.2\n  - \"We observed that the output representations ... ... (shown in figure 6(b))\": But this is also because the design of the feature extractor or the similarity measure is not suitable. Not saying GTP loss is bad, but this does not mean it is absolutely necessary. Perhaps worth mentioning it somewhere to clarify.\n- Under \"Transfer Ability Sentiment\": \"These explanation results are consistent with our.. ..happiness and joy\": I find it difficult to simply agree with. And how is it related to the proposed work? The output label distribution would follow the training data as long as the training set and evaluation set are stratified. \n", "strength_and_weaknesses": "Strength: The system and the experiment are well designed and executed. The evaluation seems to be done correctly, too. \nWeakness: The system requires a paired dataset, which is hard to find outside of classical music. The literature review lacks some relevant works. Not calling the problem music captioning may be misleading. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R8acxnvRGV", "reviewer": "Reviewer_ahJ7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the problem of generating text describing a piece of chamber music, given the acoustic features of the piece. The proposed model is relying on a text autoencoder, an audio autoencoder and a cross-modality translation model. The authors also develop the so-called group topology-preservation loss for ensuring that the representations in the music domain lie in a topology constrained by that of the text domain, which is more discriminative. The model is trained and evaluted on a dataset collected by the authors. \n\n", "summary_of_the_review": "Overall the paper is well-motivated, the main application idea is original and the execution is reasonable. However the main positive points of the paper have to do with the application itself, with little methodological contribution/analysis especially regarding the GTP loss. There is also little room for experimentation since all the experiments are done in the small collected dataset which was also used for training. Given the above application-oriented way of conducting this research, although I feel there could be audiences within ICLR finding this useful, the paper would be better suited to a more domain-specific venue. \n\nEdit after rebuttal: I thank the authors for providing detailed answers to my questions. After the rebuttal period and also seeing reviewer DjXC's concerns I would still like to keep my score. ", "strength_and_weaknesses": "\n__Strengths:__\n* Good motivation \n* New dataset\n* Reasonable learning component \n* GTP Loss \n* Results generally positive \n* Cross validation and small variance \n\n__Weaknesses:__\n* Small dataset - how to train deep learning models? \n* Little methodological contribution \n* Not entirely convinced of the GTP Loss\n* Results not very exciting and only on a single experiment/dataset  \n* Some clarity needed with presentation of scores \n* Generated sentences not always syntactically sensible, to some extent defeats the purpose of moving beyond tag generation. \n* Sentiment analysis results lack baselines \n\n__Details:__ \n\nThe motivation to extend music-based text generation beyond tags is interesting, with an underpinning effort to communicate the various nuances found in music using language, a widely understood modality for humans.\n\nThe dataset collection is a reasonable and useful step in this work. The collected dataset unfortunately is quite small and specific to a specific music type. I was actually quite surprised that the authors managed to train multi-layer, multi-modal models with less than 2K recordings. I understand a pre-trained language model was used (i.e. on English language - hope the authors can correct me if I misunderstood), would it be possible to also use a pre-trained music model instead of pre-training it on the actual collected (small) dataset? For example, by leveraging other public, even if unlabelled, music recording databases. \n\nThe learning component of the approach is quite straightforward and reasonable. The autoencoders learn representations for each modality, topped by a mapping encoder. The authors also introduce the so-called group topology-preservation (GTP) loss. I find this generally reasonable as intuition, but I'm not entirely convinced that it has been adequately justified and explored, especially being the main methodological contribution. \n\nFirstly, I wonder whether such constrain forces information in music which is not present in text to unnecesarily be aligned in a particular way, thus making the model less flexible. In other words, the GTP constrain is an indirect way of obtaining better discrimination but has additional effects which should be explored to understand them better, i.e. whether they introduce problems with generalization. Perhaps running experiments in different datasets with this loss would shed some light. \n\nSecondly, I wonder how the GTP loss compares to (cross-modality) triplet loss if we were to increase the number of negative (and perhaps also positive) samples of the triplet loss. Is it correct to say that for adequately many samples the two losses become very similar? \n\nGiven the absence of studies on the same task the authors propose a number of baselines which I find reasonable to use as competitive methods. The proposed method seems to perform well compared to tag-based approaches which is very encouraging. I had some trouble interpreting the results of the table though because I am not sure what are the minimum and maximum values of the score, e.g. the authors mention BLEU but I see scores > 1. It seems to me that the GTP loss is not introducing a very significant boost the performance. \n\nThe qualitative results seem interesting but the generated sentences are not always syntactically sensible. This to some extent defeats the purpose of moving beyond tags, but on the other hand with a larger dataset it might have been less of an issue. I also wonder if adapting large language models would solve the issue.\n\nI appreciate that the authors perform cross-validation given the small set and I think the small variance in the results is encouraging. I also enjoyed the transfer ability of sentiment, although I wonder why the results of this are not shown for the baselines. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QoTFHp2VIAq", "reviewer": "Reviewer_TFVv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel generative task of generating textual descriptions from music. They focus on a small dataset of classical music recordings paired with expert descriptions and present comparative experiments using various pre-trained language models vs their method with a novel topology-preservation loss.", "summary_of_the_review": "The shortcomings I listed above limit the work's scope. Therefore, I believe the paper is more suitable for a music information retrieval conference like ISMIR at its current stage.", "strength_and_weaknesses": "Strengths\n- The paper is well written with a few exceptions. See the specific comments below.\n- The motivation is clear.\n- The proposed methodology is sound\n- The experiments are well-prepared, however, they should be extended significantly.\n\nWeaknesses:\n\n- The expert descriptions exemplified in the paper encompass 1) quantifiable information (musical tempo, ...), 2) subjective information (valence, ...), and 3) writing style. I could also imagine that some contain 4) factual information, which is not possible to extract from only the music signal itself (e.g. name of the piece, composer). The proposed model seems to somewhat capture each element and is able to generate much more coherent examples than the other pre-trained models, but it also generates a lot of non-sensical descriptions such as \"begins with a theme of the main theme of the main theme in the...,\" \"the music is a fugue... to the sonata form of pizzicato,\" \"the music of the movement.\" \n\nI find similarities to models trained for question answering task, in particular, how LLMs fail to give correct answers - sometimes in a dangerously confident tone (e.g. \"will my stomach be cleaner if I drink bleach?\"). For me, a more interesting discussion would be about the challenges of this task, and how different models can or cannot cope with the elements mentioned above, potentially via an ablation study.\n\n- The dataset is quite small, which probably contributes to the limitations. I appreciate that it's very difficult to collect data for such a task, however, the impact will be minor unless the data could be collected in scale. For instance, the researchers could add a second dataset consisting of Anglo-american pop music (e.g. available in Lakh dataset) and crawl expert or fan reactions (e.g. from social media) and repeat the experiments. This way we could understand how generalizable the approaches are and contrast the modes of good and bad descriptions across datasets.\n\n- Classical pieces in the dataset are typically instrumental. It may not possible to draw out a universal (or a consensus between experts) sentiment from instrumental classical music (or any other type of music). The paper doesn't address subjectivity much in the account.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DEiIvTSzHPM", "reviewer": "Reviewer_DjXC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new task, music-to-text synaesthesia, in which audio features from musical tracks are extracted and interpreted as textual descriptions.  The authors collect a dataset composed of classical recordings and a set of manually annotated textual descriptions, and evaluate a multi-modal encoder-decoder model on the task.  They propose a topology preserving loss to propogate some learning signal from the text (the similarities between the text descriptions of different examples) to guide the learning of the music encoder, and show this loss improves the quality of text descriptions (measured in BLEU) over the baseline contrastive or triplet loss.\n", "summary_of_the_review": "In favor of this paper, a new dataset of paired classical music pieces and text descriptions is collected, and a number of models are evaluated on it.  It's reasonable clear in most of its presentation, and the evaluation is probably sufficiently thorough.\n\nOn the other hand, it is questionable how novel this task really is, as defined here and on this dataset, over previous music captioning and music labeling work.  There is also minimal novelty on the modeling side, really boiling down to the topology preserving loss function.  Subjectively, studying the generated descriptions did not make a compelling case for the descriptions being more informative than having a small set of text labels (albeit necessary to have more than the single labels gleaned from the music categories).  I found the general motivation of the task confusing and overlooking previous related work.", "strength_and_weaknesses": "The authors begin by laying out an awkward case, defining translating information between modalities as synaesthesia, and therefore claiming that speech recognition and image captioning as forms of synaesthesia.  I don't want to argue the particular semantics of this, only to say that if synaesthesia here is going to be applied so broadly, it loses any important meaning and misleads the reader.  Being familiar with the definitions of synaesthesia as a -perceptual- phenomenon, I certainly thought there would be deeper cognitive connections, so felt a bit mislead by the title. Alternatively, in terms of modeling, I would have expected something more along the lines of [2], where more emphasis is put on a shared latent space (not unlike earlier sorts of image captioning - Socher, etc.).  \n\nBut if image captioning is a form of synaesthesia, then there is another name for such research that springs to mind: music captioning.  Looking down this line of research, a number of related work pops up [1-4].  And in terms of music labeling vs. music \"synaesthesia\", if the only difference between these is whether an output vector is mapped to one label from a set of many, or decoded into many words, it seems a weak point from which to pitch this research as having an important distinction from other related audio->text tasks.  So while I think the general idea of learning to predict textual descriptions from raw music audio is interesting and should be pursued, the presentation of this paper falls short in terms of recognizing existing work, or establishing a novel task.\n\nIt terms of the dataset, it seems useful but is limited in both size and the domain of being a specific set of classical music pieces.  On the text side, the descriptions (as viewed through the generating text descriptions) do not seem that diverse nor bring something by virtue of being full sentences that could not have been conveyed by bag of words.\n\nThe experimental setup is reasonable given the lack of established baselines for this type of task, yet, they also seem weak enough or inherently disadvantaged that they serve little competitive function.  The two experiments that are more interesting are with/without the topographic loss modifications.  The GTP loss itself seems well-motivated and may be only applicable to datasets of this sort, but stands out as one of the novel contributions.  However, when we see the generated text descriptions in Table 2, it led me to believe the learning problem on this data is quite simple, and might be the classification of just a few categories, strung together with text, and much less of a newly established task where the text descriptions themselves offer something qualitatively more informative than a multi-label classification task.\n\n\n[1]\n@article{Choi2016TowardsMC,\n  title={Towards Music Captioning: Generating Music Playlist Descriptions},\n  author={Keunwoo Choi and Gy{\\\"o}rgy Fazekas and Mark B. Sandler},\n  journal={ArXiv},\n  year={2016},\n  volume={abs/1608.04868}\n}\n\n[2]\n@article{Manco2022ContrastiveAL,\n  title={Contrastive Audio-Language Learning for Music},\n  author={Ilaria Manco and Emmanouil Benetos and Elio Quinton and Gy{\\\"o}rgy Fazekas},\n  journal={ArXiv},\n  year={2022},\n  volume={abs/2208.12208}\n}\n\n[3]\n@article{Manco2021MusCapsGC,\n  title={MusCaps: Generating Captions for Music Audio},\n  author={Ilaria Manco and Emmanouil Benetos and Elio Quinton and Gy{\\\"o}rgy Fazekas},\n  journal={2021 International Joint Conference on Neural Networks (IJCNN)},\n  year={2021},\n  pages={1-8}\n}\n\n[4]\nAnd arguably:\n@article{Gao2022MusicQA,\n  title={Music Question Answering:Cognize and Perceive Music},\n  author={Wenhao Gao and Xiaobing Li and Cong Jin and Tie Yun},\n  journal={2022 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)},\n  year={2022},\n  pages={1-6}\n}\n\n\nEquation (4), BLEU vs. later BLUE.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject