Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of generating text describing a piece of chamber music, given the acoustic features of the piece. The proposed model is relying on a text autoencoder, an audio autoencoder and a cross-modality translation model. The authors also develop the so-called group topology-preservation loss for ensuring that the representations in the music domain lie in a topology constrained by that of the text domain, which is more discriminative. The model is trained and evaluted on a dataset collected by the authors. 


strength_and_weaknesses: 
__Strengths:__
* Good motivation 
* New dataset
* Reasonable learning component 
* GTP Loss 
* Results generally positive 
* Cross validation and small variance 

__Weaknesses:__
* Small dataset - how to train deep learning models? 
* Little methodological contribution 
* Not entirely convinced of the GTP Loss
* Results not very exciting and only on a single experiment/dataset  
* Some clarity needed with presentation of scores 
* Generated sentences not always syntactically sensible, to some extent defeats the purpose of moving beyond tag generation. 
* Sentiment analysis results lack baselines 

__Details:__ 

The motivation to extend music-based text generation beyond tags is interesting, with an underpinning effort to communicate the various nuances found in music using language, a widely understood modality for humans.

The dataset collection is a reasonable and useful step in this work. The collected dataset unfortunately is quite small and specific to a specific music type. I was actually quite surprised that the authors managed to train multi-layer, multi-modal models with less than 2K recordings. I understand a pre-trained language model was used (i.e. on English language - hope the authors can correct me if I misunderstood), would it be possible to also use a pre-trained music model instead of pre-training it on the actual collected (small) dataset? For example, by leveraging other public, even if unlabelled, music recording databases. 

The learning component of the approach is quite straightforward and reasonable. The autoencoders learn representations for each modality, topped by a mapping encoder. The authors also introduce the so-called group topology-preservation (GTP) loss. I find this generally reasonable as intuition, but I'm not entirely convinced that it has been adequately justified and explored, especially being the main methodological contribution. 

Firstly, I wonder whether such constrain forces information in music which is not present in text to unnecesarily be aligned in a particular way, thus making the model less flexible. In other words, the GTP constrain is an indirect way of obtaining better discrimination but has additional effects which should be explored to understand them better, i.e. whether they introduce problems with generalization. Perhaps running experiments in different datasets with this loss would shed some light. 

Secondly, I wonder how the GTP loss compares to (cross-modality) triplet loss if we were to increase the number of negative (and perhaps also positive) samples of the triplet loss. Is it correct to say that for adequately many samples the two losses become very similar? 

Given the absence of studies on the same task the authors propose a number of baselines which I find reasonable to use as competitive methods. The proposed method seems to perform well compared to tag-based approaches which is very encouraging. I had some trouble interpreting the results of the table though because I am not sure what are the minimum and maximum values of the score, e.g. the authors mention BLEU but I see scores > 1. It seems to me that the GTP loss is not introducing a very significant boost the performance. 

The qualitative results seem interesting but the generated sentences are not always syntactically sensible. This to some extent defeats the purpose of moving beyond tags, but on the other hand with a larger dataset it might have been less of an issue. I also wonder if adapting large language models would solve the issue.

I appreciate that the authors perform cross-validation given the small set and I think the small variance in the results is encouraging. I also enjoyed the transfer ability of sentiment, although I wonder why the results of this are not shown for the baselines. 

clarity,_quality,_novelty_and_reproducibility: The paper is relatively well written but I have some comments/questions which follow in the end of this section, I think addressing those would make the paper much clearer. 

The paper is novel but quite application oriented, and explores only a small dataset. 

The authors have provided code. 

__Some comments / questions for clarity__ 
 
-  I am not sure whether introducing the term "synaesthesia" helps or whether it confuses the reader, since such tasks are already known in the literature (e.g. captioning for images). Nevertheless, as far as I'm aware this task has not been tackled for music, although I am not very familiar with the applications of generative models in the music domain.  
- Three sentences before sec. 3: Can you elaborate, I don't fully follow. 
- It would help to mention earlier on in the text that the raw music spectrum used also captures some temporal aspects as it is made clearer in "Implementation"   
- Second bullet in "Comparison models": Can you be a bit clearer what are the pre-trained parts of those methods and where / how they have been pre-trained? 
- Second sentence in 4.2: Can you clarify further? I am not sure I understand what "low performance" refers to and what is the referred "increased difficulty for humans...". 
- Table 1: What are the limits (smallest/largest possible) for the scores presented here?  

summary_of_the_review: Overall the paper is well-motivated, the main application idea is original and the execution is reasonable. However the main positive points of the paper have to do with the application itself, with little methodological contribution/analysis especially regarding the GTP loss. There is also little room for experimentation since all the experiments are done in the small collected dataset which was also used for training. Given the above application-oriented way of conducting this research, although I feel there could be audiences within ICLR finding this useful, the paper would be better suited to a more domain-specific venue. 

Edit after rebuttal: I thank the authors for providing detailed answers to my questions. After the rebuttal period and also seeing reviewer DjXC's concerns I would still like to keep my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a novel generative task of generating textual descriptions from music. They focus on a small dataset of classical music recordings paired with expert descriptions and present comparative experiments using various pre-trained language models vs their method with a novel topology-preservation loss.
strength_and_weaknesses: Strengths
- The paper is well written with a few exceptions. See the specific comments below.
- The motivation is clear.
- The proposed methodology is sound
- The experiments are well-prepared, however, they should be extended significantly.

Weaknesses:

- The expert descriptions exemplified in the paper encompass 1) quantifiable information (musical tempo, ...), 2) subjective information (valence, ...), and 3) writing style. I could also imagine that some contain 4) factual information, which is not possible to extract from only the music signal itself (e.g. name of the piece, composer). The proposed model seems to somewhat capture each element and is able to generate much more coherent examples than the other pre-trained models, but it also generates a lot of non-sensical descriptions such as "begins with a theme of the main theme of the main theme in the...," "the music is a fugue... to the sonata form of pizzicato," "the music of the movement." 

I find similarities to models trained for question answering task, in particular, how LLMs fail to give correct answers - sometimes in a dangerously confident tone (e.g. "will my stomach be cleaner if I drink bleach?"). For me, a more interesting discussion would be about the challenges of this task, and how different models can or cannot cope with the elements mentioned above, potentially via an ablation study.

- The dataset is quite small, which probably contributes to the limitations. I appreciate that it's very difficult to collect data for such a task, however, the impact will be minor unless the data could be collected in scale. For instance, the researchers could add a second dataset consisting of Anglo-american pop music (e.g. available in Lakh dataset) and crawl expert or fan reactions (e.g. from social media) and repeat the experiments. This way we could understand how generalizable the approaches are and contrast the modes of good and bad descriptions across datasets.

- Classical pieces in the dataset are typically instrumental. It may not possible to draw out a universal (or a consensus between experts) sentiment from instrumental classical music (or any other type of music). The paper doesn't address subjectivity much in the account.
clarity,_quality,_novelty_and_reproducibility: The writing is mostly clear and of good quality. Below I list some issues I noticed:

- "convert Italic, French, German" -> "Italian"
- the first paragraph of "Preliminary Exploration" is written too informally.
- I am not sure why Related work is in Section 5 and not earlier in the paper.
- Missing relevant work: Won, M., Salamon, J., Bryan, N. J., Mysore, G. J., & Serra, X. (2021). Emotion Embedding Spaces for Matching Music to Stories. ISMIR 2021 

The task is novel.

The code is given online, but the data isn't. I assume everything will be open after acceptance.
summary_of_the_review: The shortcomings I listed above limit the work's scope. Therefore, I believe the paper is more suitable for a music information retrieval conference like ISMIR at its current stage.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new task, music-to-text synaesthesia, in which audio features from musical tracks are extracted and interpreted as textual descriptions.  The authors collect a dataset composed of classical recordings and a set of manually annotated textual descriptions, and evaluate a multi-modal encoder-decoder model on the task.  They propose a topology preserving loss to propogate some learning signal from the text (the similarities between the text descriptions of different examples) to guide the learning of the music encoder, and show this loss improves the quality of text descriptions (measured in BLEU) over the baseline contrastive or triplet loss.

strength_and_weaknesses: The authors begin by laying out an awkward case, defining translating information between modalities as synaesthesia, and therefore claiming that speech recognition and image captioning as forms of synaesthesia.  I don't want to argue the particular semantics of this, only to say that if synaesthesia here is going to be applied so broadly, it loses any important meaning and misleads the reader.  Being familiar with the definitions of synaesthesia as a -perceptual- phenomenon, I certainly thought there would be deeper cognitive connections, so felt a bit mislead by the title. Alternatively, in terms of modeling, I would have expected something more along the lines of [2], where more emphasis is put on a shared latent space (not unlike earlier sorts of image captioning - Socher, etc.).  

But if image captioning is a form of synaesthesia, then there is another name for such research that springs to mind: music captioning.  Looking down this line of research, a number of related work pops up [1-4].  And in terms of music labeling vs. music "synaesthesia", if the only difference between these is whether an output vector is mapped to one label from a set of many, or decoded into many words, it seems a weak point from which to pitch this research as having an important distinction from other related audio->text tasks.  So while I think the general idea of learning to predict textual descriptions from raw music audio is interesting and should be pursued, the presentation of this paper falls short in terms of recognizing existing work, or establishing a novel task.

It terms of the dataset, it seems useful but is limited in both size and the domain of being a specific set of classical music pieces.  On the text side, the descriptions (as viewed through the generating text descriptions) do not seem that diverse nor bring something by virtue of being full sentences that could not have been conveyed by bag of words.

The experimental setup is reasonable given the lack of established baselines for this type of task, yet, they also seem weak enough or inherently disadvantaged that they serve little competitive function.  The two experiments that are more interesting are with/without the topographic loss modifications.  The GTP loss itself seems well-motivated and may be only applicable to datasets of this sort, but stands out as one of the novel contributions.  However, when we see the generated text descriptions in Table 2, it led me to believe the learning problem on this data is quite simple, and might be the classification of just a few categories, strung together with text, and much less of a newly established task where the text descriptions themselves offer something qualitatively more informative than a multi-label classification task.


[1]
@article{Choi2016TowardsMC,
  title={Towards Music Captioning: Generating Music Playlist Descriptions},
  author={Keunwoo Choi and Gy{\"o}rgy Fazekas and Mark B. Sandler},
  journal={ArXiv},
  year={2016},
  volume={abs/1608.04868}
}

[2]
@article{Manco2022ContrastiveAL,
  title={Contrastive Audio-Language Learning for Music},
  author={Ilaria Manco and Emmanouil Benetos and Elio Quinton and Gy{\"o}rgy Fazekas},
  journal={ArXiv},
  year={2022},
  volume={abs/2208.12208}
}

[3]
@article{Manco2021MusCapsGC,
  title={MusCaps: Generating Captions for Music Audio},
  author={Ilaria Manco and Emmanouil Benetos and Elio Quinton and Gy{\"o}rgy Fazekas},
  journal={2021 International Joint Conference on Neural Networks (IJCNN)},
  year={2021},
  pages={1-8}
}

[4]
And arguably:
@article{Gao2022MusicQA,
  title={Music Question Answering:Cognize and Perceive Music},
  author={Wenhao Gao and Xiaobing Li and Cong Jin and Tie Yun},
  journal={2022 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)},
  year={2022},
  pages={1-6}
}


Equation (4), BLEU vs. later BLUE.
clarity,_quality,_novelty_and_reproducibility: The presentation was reasonably clear and there would not be too much difficulty in reproducing most of the results.  However it is a bit lacking in terms of novelty and substance.
summary_of_the_review: In favor of this paper, a new dataset of paired classical music pieces and text descriptions is collected, and a number of models are evaluated on it.  It's reasonable clear in most of its presentation, and the evaluation is probably sufficiently thorough.

On the other hand, it is questionable how novel this task really is, as defined here and on this dataset, over previous music captioning and music labeling work.  There is also minimal novelty on the modeling side, really boiling down to the topology preserving loss function.  Subjectively, studying the generated descriptions did not make a compelling case for the descriptions being more informative than having a small set of text labels (albeit necessary to have more than the single labels gleaned from the music categories).  I found the general motivation of the task confusing and overlooking previous related work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a music captioning system that was trained on a classical music dataset. A modification on loss is proposed to improve the performance. 
strength_and_weaknesses: Strength: The system and the experiment are well designed and executed. The evaluation seems to be done correctly, too. 
Weakness: The system requires a paired dataset, which is hard to find outside of classical music. The literature review lacks some relevant works. Not calling the problem music captioning may be misleading. 
clarity,_quality,_novelty_and_reproducibility: Clarify and quality: Paper is well written, experiment and evaluation are clear. The justification of GTP can be clarified better. 
Novelty: Medium. Treating music captioning (or any captioning) as sequence-to-sequence (i.e., translation) problem exists. The proposed loss is interesting but of limited novelty.
Reproducibility: The proposed model is explained well but not completely for those who want to re-implement it. 
summary_of_the_review: Title - Is there any reason to avoid music captioning?
Introduction - Similarly, the recent progress in music captioning (and even audio captioning maybe) is missing.
Section 2 - I appreciate the preliminary exploration. 
Section 3 - Good approach to see the problem a cross-modality translation problem. The proposed system and GTP loss seems sensible to me.
On GTP section 
  - "by the autoencoder suffer from high non-discrimination.": Can you elaborate this?
  - "Due to the relatively high discrimination in.. ..generation.": I get it, but it'd be nicer if this is elaborated, too.
- Is "node" a music item?
Section 4
 - Choice of BLEU - It makes sense. But it would be even better if there's some example about how BLEU works for this problem / dataset.
Section 4.2
  - "We observed that the output representations ... ... (shown in figure 6(b))": But this is also because the design of the feature extractor or the similarity measure is not suitable. Not saying GTP loss is bad, but this does not mean it is absolutely necessary. Perhaps worth mentioning it somewhere to clarify.
- Under "Transfer Ability Sentiment": "These explanation results are consistent with our.. ..happiness and joy": I find it difficult to simply agree with. And how is it related to the proposed work? The output label distribution would follow the training data as long as the training set and evaluation set are stratified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper considers the following research problem of generating descriptive texts from music recordings. Given that this task is beyond the capability of the current existing music-related datasets, the authors collect a new dataset which contains paired classical music recordings and text descriptions. The proposed model is a cross-modality translation model, which consists of a text autoencoder and an audio autoencoder. A group topology-preservation (GTP) loss is developed for employing the text (which is more discriminative) to guide the music representation generation. Experimental results are presented on the collected dataset. 

Strengths: All the reviewers agree that the motivation of the paper is very interesting and it is great to have a new dataset for this particular problem. Overall the proposed method makes intuitive sense (albeit still with some confusions on execution as pointed out by reviewer ahJ7). 

Weaknesses: There are a few major weaknesses which make accepting this paper harder to justify: 1) Currently the paper has relatively little methodological contribution. As reviewer ahJ7 pointed out, the proposed GTP loss makes intuitive sense, but is not adequately justified and explored; 2) As multiple reviewers pointed out, the dataset collected by the authors is quite small and it consists entirely of classical music which limits its applicability; 3) Multiple reviewers found the use of the term "synaesthesia" misleading given that this paper can also be descrbied as "music captioning" where some existing literature can also be included/considered. 4) The experimental setup seems reasonable but the baselines all seem pretty weak to offer much meaningful information. 


justification_for_why_not_higher_score: Based on the reviewers' comments, I think it is very much unanimously agreed that this paper still needs a bit more work before it's ready for publication. One reviewer also suggests that a more domain-specific venue might be a good option.
Decision: Reject