Audio Prototypical Network for Controllable Music Recommendation

Authors: Fırat Öncel, Emiliano Penaloza, Haolun Wu, Shubham Gupta, Mirco Ravanelli, Laurent Charlin, Cem Subakan
Primary area: interpretability and explainable AI
Keywords: Scrutable Recommender System, Controllability, Explainable Machine Learning

Abstract:
Traditional recommendation systems represent user preferences in dense representations obtained through black-box encoder models. While these models often provide strong recommendation performance, they lack interpretability for users, leaving users unable to understand or control the system’s modeling of their preferences. This limitation is especially challenging in music recommendation, where user preferences are highly personal and often evolve based on nuanced qualities like mood, genre, tempo, or instrumentation. 
    In this paper, we propose an audio prototypical network for controllable music recommendation. This network expresses user preferences in terms of prototypes representative of semantically meaningful features pertaining to musical qualities. We show that the model obtains competitive recommendation performance compared to popular baseline models while also providing interpretable and controllable user profiles.

Decision: reject
Reviewer scores: 3, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729047343227

thread_type:
official_review

<thread_object>
creator: reviewer_pdqh

<content>
summary:
The paper proposes a "prototypical network" for music recommendation, which basically expresses user preferences in terms of a combination of listenable prototypes (i.e., short musical snippets). This is argued to be a more interpretable and controllable form of music recommendation as the tags (and snippets based on them) are interpretable and weighted preferences are controllable by the user.

strengths:
The idea of using listenable prototypes to improve the interpretability of music recommendation is (to me) original, at least in the context of music recommendation.

weaknesses:
Ultimately the prototypes did not strike me as convincing, and I didn't see additional value in them compared to more traditional tags. The generated audio did not seem to be faithful to the tags (e.g. music with male vocalist didn't have any vocalist -- this was just the first one I tried). So I'm not really convinced that "listenability" really adds to the usefulness of the method compared to a (presumably more trivial) approach purely based on tags. I'm not sure I generally see the value of a generative approach in this context, compared to e.g. extractively choosing snippets from songs a user likes. As a user it would be difficult for me to express my preferences in terms of these listenable snippets which often did not sound like "real" music.

Performance is also probably a sticking point. The authors acknowledge that the method is competitive with, but not stronger than baseline methods, and offer increased controllability to overcome this shortcoming. But I think that requires that the controllability / interpretability claims are totally convincing, which they aren't quite yet.

questions:
-- Why does the approach need to be generative (abstractive) rather than extractive?

-- Is it possible to more directly compare against the strongest baselines on MSD? I understand there were some issues here since not all the songs contain audio and the dataset had to be sampled (and baselines rerun) but I am maybe missing a summary of whether the performance is really close to SotA for this dataset.

-- From the user perspective, what is the value of this system compared to just surfacing the tags themselves? To the extent that the tags *don't* match the snippets, the system isn't faithful and will confuse users. To the extent that the tags *do* match the snippets, the snippets are (arguably) redundant.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651186525

thread_type:
official_review

<thread_object>
creator: reviewer_eea9

<content>
summary:
This paper proposes a prototype-based network for explainable recommendations. The key model pipeline involves (1) prototypes generated by a controllable music generation model (MusicGen), (2) transformed user history into an interpretable sequence, where multi-head attention was used to learn the weight distributions over prototypes, and (3) a feed-forward neural network for the recommendation task as an extreme classification problem, where two auxiliary objectives accompany the recommendation objective: a controllability objective to impose calibration of user-level tag preference, and a prototype-separability objective to avoid prototype collapse. 

The experiments were conducted on MSD, a commonly used music recommendation dataset. The paper claims improvements in controllability through their proposed metric, which is calculated based on the difference of tag-wise ranking after removing certain prototypes, while maintaining competitive performance to the compared recommendation baselines.

strengths:
This paper introduces an interesting perspective on combining prototype networks and the controllability of recommendations. The fact that the prototypes were interpretable (i.e. listenable music clips) is a nice feature, and the controllability is measured by the calibration of user tag preferences also provides a direct way to implement user controls. 

The experiments seem to indicate the effectiveness of the added two objectives by improving the recommendation performance. The authors further analyzed the tag-wise performance drop to support the importance of these prototypes. A series of ablation studies were conducted to study the importance of model parameters.

weaknesses:
The method lacks novelty: each component of the whole model is not new. The key concept of using prototypes for explainable recommendations has been explored in [1]. Different from [1], the number of prototypes is fixed in this paper and aligned with pre-defined song tags, which can limit the expressiveness of the model and may suffer from noisiness in tag data. The quality of these prototypes is delegated to a generative music model, but the experiments do not address details on how the quality may affect model training and controllability. 

The baselines compared is rather limited: To show the recommendation performance is competitive, the authors shall also compare models with reported superior performance on the MSD dataset such as [2] (as reported in the RecVAE paper) to provide more references. I also recommend the authors consider comparing to ProtoMF [1], since method-wise they also used the concept of prototype network for recommendations.

[1] Alessandro B. Melchiorre, Navid Rekabsaz, Christian Ganhör, and Markus Schedl. 2022. ProtoMF: Prototype-based Matrix Factorization for Effective and Explainable Recommendations. In Proceedings of the 16th ACM Conference on Recommender Systems (RecSys '22). ACM, 246–256. 
[2] Harald Steck. 2019. Embarrassingly Shallow Autoencoders for Sparse Data. In The World Wide Web Conference. ACM, 3251–3257.

questions:
1. Can you explain more about the process of prototype generation, in terms of why choosing a certain generative model, the diversity (similarity between generated prototypes), and quality (tag-music alignment) of the generated prototypes? Furthermore, how it may affect the performance of the resulting recommendation model? 

2. Can you further elaborate on the ablation study on training loss components (sec 5.4), in particular, was the monotonically increasing recommendation performance suggesting the recommendation model without the regularization terms was badly trained? If you add more searches on $\lambda_{1}$ and $\lambda_{2}$, will you see a tradeoff between the multiple objectives?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684377175

thread_type:
official_review

<thread_object>
creator: reviewer_kz1t

<content>
summary:
This paper proposes a music recommendation method based on audio prototypes. The method first inputs the music tags into the music generation model to generate the corresponding audio; then the MERT model is used to extract the final feature of 1024 dimensions of each generated audio. During the training process, each song in the user's behavior obtains the weighted representation of tag-audios through attention network; the user's representation is the sum of all behaviors(songs) and used for recommendation in final.

strengths:
1. This paper attempts to use audio features for music recommendation to solve the problem of insufficient keywords/tags problem.
2. This paper proposes a metric to test the controllability of the model.

weaknesses:
1. Although the author proposes that the model is controllable, the review is not clear about the definition of controllability in the paper, and the comparison between the model and other baseline methods (only one), and where its controllability is superior.
2. There are many classic CF methods and DL-based methods for recommendation based on user behavior. The reviewer noticed that the paper only selected VAE-based methods for comparison without any explanation or motivation of such selection.
3. The paper emphasizes the use of prototype audio of music for recommendation. These audios are generated by the original tags of the music via MusicGen, but the author does not explain the details and settings of the generation process, not provide analysis and distribution of the generated audio, and cannot provide proof of whether there is an essential difference between these audios and the original tags in the role of recommendation, nor does it provide comparative experiments to prove the difference between the two. Therefore, the review cannot determine that the paper solves the limitation problem of tag-based recommendation proposed by audio-based recommendation.

questions:
The main questions are the problems listed in the weakness. In addition, there is a technical question in line140 equation 3 and line148 equation 4. Are the  $x^i_j$  in the two equations the same or not? How is the query $x^i_j$ in the attention represented?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731036986046

thread_type:
official_review

<thread_object>
creator: reviewer_s6hu

<content>
summary:
This paper aims to tackle the task of controllable music recommendation. It leverage music clip level prototypes as explanations and propose an attention-based recommendation model to fulfil the task. Experiments are conducted to demonstrate how this method performs the control during recommendation. Multiple experiments illustrate several properties of this method.

strengths:
* The task of controllable music recommendation is valuable in both academia and industry. 
* The motivation of using music-clip level prototypes is reasonable and clear. And the way to directly use music content for recommendation is a promising direction.
* The writing is clear and easy to follow.

weaknesses:
1. Less technical novelty: 
* The proposed prototype-based controllable music recommender model is a quite straightforward attention-based neural network model with certain losses. The attention-based model architecture has been proposed and extensively studied in recommender systems, which even though is practical and helpful,  it is not quite novel to the research or industry community. 
* The learning or extraction of the prototype is based on some existing methods (MERT or MusicGen). I am expecting certain innovations in this part, which is quite interesting. For example, is it possible to automatically mine such prototypes purely based on user behaviours’ data as supervision? Furthermore, the music understanding model that is used to extract such prototypes should also be optimized during this process (in either end2end or multi-stage manners)?   

2. Lacking rigorous evaluation:
* Only one dataset. I understand that it is not easy to obtain many datasets pertaining to this task formulation, while only using one dataset is less convincing to justify the generalization capability of the proposed method.
* The baselines are just general music recommendation models, which could be weak. Explainable recommendation is a quite popular topic in the past few years, and I assume there should be quite a number of works that can be implemented or adapted to this task, while they are not included in this paper. In addition to performance comparison with baselines, the controllability comparison with baselines should be also considered. 
* For Section 5.5, there are many cold-start recommendation methods that can be implemented as baselines. I hope you can select some methods to do a comparison study instead of only comparing with the model itself in the non-cold start setting. 
* Even though the target of this paper is the controllability rather than overall performance, it is still a concern that the quantitative performance of the model is not very strong.

questions:
Please see the above comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes the audio prototypical network method for controllable music recommendation. The motivation to improve the interpretability and controllability of music recommendations is recognized by the reviewers, and the paper is easy to understand. However, reviewers also pointed out some key weaknesses, including: 1) Less technical novelty, as attention-based neural networks can achieve similar results; 2) Insufficient experiments, especially in terms of baseline selection and dataset (currently only one dataset is used); furthermore, there is controversy over the controllability metrics, and the final model's accuracy did not outperform existing models; 3) Unclear differences with existing settings. Since there is only one template per tag, the system is not significantly different from tag-based prototypes. As a result, all reviewers believe the paper is not ready for publication by far.