Diversity-Rewarded CFG Distillation
Authors: Geoffrey Cideron
Andrea Agostinelli
Johan Ferret
Sertan Girgin
Romuald Elie
Olivier Bachem
Sarah Perrin
Alexandre Rame
Venue: ICLR 2025 Poster (2025)
Keywords: Music Generation
Distillation
CFG
RL
Diversity
Model Merging
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Generative models are transforming creative domains such as music generation, with inference-time strategies like Classifier-Free Guidance (CFG) playing a crucial role. However, CFG doubles inference cost while limiting originality and diversity across generated contents. In this paper, we introduce diversity-rewarded CFG distillation, a novel finetuning procedure that distills the strengths of CFG while addressing its limitations. Our approach optimises two training objectives:  (1) a distillation objective, encouraging the model alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL objective with a diversity reward, promoting the generation of diverse outputs for a given prompt. By finetuning, we learn model weights with the ability to generate high-quality and diverse outputs, without any inference overhead. This also unlocks the potential of weight-based model merging strategies: by interpolating between the weights of two models (the first focusing on quality, the second on diversity), we can control the quality-diversity trade-off at deployment time, and even further boost performance. We conduct extensive experiments on the MusicLM text-to-music generative model, where our approach surpasses CFG in terms of quality-diversity Pareto optimality.  According to human evaluators, our finetuned-then-merged model generates samples with higher quality-diversity than the base model augmented with CFG. Explore our generations at https://musicdiversity.github.io/.

Published as a conference paper at ICLR 2025
DIVERSITY-REWARDE CFG DISTILLATION
Geoffrey Cideronī Andrea Agostinelli, Johan Ferret, Sertan Girgin, Romuald Elie Olivier Bachem, Sarah Perrin*, Alexandre Ramé*
Google DeepMind, * Equal advisory contribution
ABSTRACT
Generative models are transforming creative domains such as music generation, with inference- strategies like Classifier-Fres Guidance (CFG) playing a crucial role. However, CFG doubles inference cost while limiting originality and diversity across generated contents. In this paper, introduce diversity- -rewarded CFG distillation, novel finetuning procedure that distills the strengths of CFG while addressing its limitations. Our approach optimises two training objectives: (1) a distillation objective, encouraging the model alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL objective with diversity reward, pro- moting the generation of diverse outputs for a given prompt. By finetuning, we learn model weights with the ability to generate high-quality and diverse outputs, without any inference overhead. This also unlocks the potential of weight-based model merging strategies: by interpolating between the weights of two models (the first focusing quality, the second on diversity), we can control the quality-diversity trade-off at deployment time, and even further boost performance. We conduct extensive experiments on the Mus i eLM text-to-musii generative model, where our approach surpasses CFG in terms of quality- to human evaluators, our finetuned-then-mreye higher quality-diversiy than the base model augmented with CFG. Explore our generations at google-research. git.ab.io/eenne///sscci//iiiiiie mmssc..
Pareto optimality. According model generates samples with
INTRODUCTION
Generative models for creative domains. Art and entertainment - domains historically driven by human creativity - are undergoing a profound transformation thanks to Al generative models. These models, often powered by Large Language Models (LLMs) or diffusion, can now generate texts (Gemini Team, 2023), images (Ramesh al., 2022), videos (Ho et al., 2022), and audios (Borsos et al., 2023; Kreuk et al., 2023; Copet et al., 2023; Agostinelli et al., 2023; Cideron et al., 2024; Défossez et al.). To further refine the quality, these models are often augmented with inference methods during weal-world deployment, ranging from simple temperature scaling more refined methods like Beam search (Freitag & Al-Onaizan, 2017), test-time augmentation (Shanmugam et al., 2021), or MCTS (Kocsis & Szepesvári, 2006). A particularly popular method for image and audio generation is classifier-free guidance (CFG) (Ho & Salimans, 2022), e.g. used in DALL-E (Ramesh al., 2022) or AudioGen (Kreuk al., 2023). CFG improves the model's fidelity the prompt by combining the logits of conditional and unconditional generations. Despite its benefits, CFG has two main limitations: doubles the computational cost during deployment and reduces the diversity of generated content (Ho & Salimans, 2022; Dhariwal & Nichol, 2021; Kreuk et al., 2023; Meng et al., 2023), hindering the exploration of novel and diverse ideas a cornerstone of creativity. Ideally, these models should not only fulfill user intent but also surprise them with unexpected and innovative outputs: the model should not generate systematically the same content (Hamilton, 2024).
Quality-diversin trade-off. Effectively controlling the quality-diversity trade-off is thus extremely important but challenging. On the one hand, optimising quality usually reduces diversity; this limitation affects inference methods such CFG but also finetuning stages such RLHF (Kirk et al., 2024; Mohammadi, 2024). On the other hand, promoting diversity usually reduces quality (Brown
1 Correspondence Geoffrey Cideron <gei deiiddrrnnggogg e

---

Published as a conference paper at ICLR 2025
CFG distillations
minimise KL between logits
Logis
0.65
Prompt
Generation
Student
KL
0.60
CFG log
0.55
Teacher
Diversity
0.50
Diversity-rewardee
maximise diversity reward
Geraration
0.45
6-0
Y=2
B-5
Prompt
Diversity
0.40
B=10
y=
B=
LERPIO.
15)
CFG
Diversity-rewarde CFG distillation objective =- KL Diversity
3.8
4.0
4.2
4.4
4.5
Quality
Figure/ 1: Left. Illustration of the two objectives: CFG distillation (above) and the diversity reward (below), multiplied by the diversity coefficient in the joint finetuning objective. Right. Quality- diversity trade-off for different strategies. The first four lines represent the training trajectories of our approach, distilling CFG (with y = 3) with varying diversity coefficient / in {0, 5, 10, 15); every 500 training steps, evaluate the quality and diversity of the generations. Larger values of 9 lead to more diverse models yet slightly less quality. For linear interpolation (LERP), each cross corresponds to 0 < X I 1 when interpolating between the weights 0g of quality-focused model (B = 0) and those 0g of a diversity-
model (3 = 15); the evaluated generations are obtained from the weights A) - 0g + À Og. For the CFG baseline, each dot corresponds
different value for the guidance factor 1 S < 7. This plot shows that our method improves the quality-diversity trade-off: notably, LERP uncovers strong and steerable front of solutions by just interpolating between the weights of two models, at deployment time.
et al., 2005), e.g. when increasing temperature at inference (Zhang et al., 2021), As further discussed in Section 4, quality-diversity algorithms (Lehman & Stanley, 2011; I; Mouret & Clune, 2015) seek to train a population of models with diverse abilities. In contrast, optimising directly the diversity of generations of single model is an under-explored yet promising avenue, that we investigate here.
rewaadd CFG distillation. In this work, we introduce a novel finetuning strategy to enhance the quality- diiverity trade-off generative models for creative domains. Specifically, we combine distillation and reinforcement learning (RL) to optimise two complementary objectives. The first is a novel CFG distillation objective for LLMs where we distill the behavior of teacher into the student. Critically, the teacher is the CFG-augmented base model, rather than a larger third-party model as commonly done the literature (Hinton et al., 2015). This involves minimizing the KL divergence between the logits of the teacher and the student the data distribution generated by the student (to reduce train-test mismatch), following the on-policy distillation framework of Agarwal et al. (2024). By distilling CFG into the model weights, eliminating CFG's ' inference overhead. The second is a novel RL. with a diversity reward objective, maximising the diversity across pairs generations for given prompt. Diversity measured with by comparing pairs of generations by first embedding them and then computing their negative (cosine) similarity in this embedding space. Combining these objectives allows the finetuned model to inherit the quality of CFG without its cost, while simultaneously maintaining diversity through the RL objective, thus solving the two main drawbacks of CFG at inference time.
improve generation quality while
Model merging, The hyperparameter 3 (multiplier for the diversity reward) allows controlling the quality-diversity trade-off training time, as shown Figure (right); in contrast, traditional CFG can do it deployment time. To enable this level of control within our approach, we propose third contribution involving model merging. Specifically, we finetune two models, one focusing on quality (low 3) and the other focusing diversity (high 3), and then combine their weights by linear interpolation (LERP) (Utans, 1996), balancing between quality and diversity. This follows from the linear mode connectivity property (Frankle et al., 2020) and recent advances in model merging (Wortsman et al., 2022a; Ramé et al., 2022; Ilharco et al., 2023), showing that weights finetuned from shared pretrained initialisation be interpolated, despite the coneriirasssies in the architecture.
2

---

Published as a conference paper at ICLR 2025
Notably, Ramé et al. (2023) showed that interpolating between weights finetumed different rewards trades their abilities off. Thus, we interpolate between the weights of quality- -ocused model and a peerssty-focuset model: sliding the interpolating coefficient À between 0 and 1 uncovers strong and steerable quality-diverse front of solutions, without overhead at deployment. Crucially, we find that the interpolated model using À 0.5 is our best model, outperforming models explicitly finetuned for intermediate values of the diversity hyperparameter B.
Contribution 1: CFG distillation for quality in LLMs. We distill the quality benefits of costly anference-time strategy, CFG, into the weights of our model.
Contribution 2: Reinforcement learning for diversity. We reward the model to create diverse generations, reducing the drop in diversity caused by CFG and finetuning.
Model merging for Pareto-optimality. We trade-off quality and diversity at deployment time by interpolating between the weights of a quality- model and diversity- model.
Contribution 3:
Music generation. We apply strategy to text-to-music generation, a creative task where balancing quality and diversity key. Specifically, we finetune Mus cLM (Agostinelli et al., 2023) and consistently improve the quality-diversity trade-off achieved by the CFG-augmented previous state- of-the-art (Cideron et al., 2024). Our experiments, featuring human evaluations, validate that our models generate more diverse music while maintaining high quality.
2 DIVERSITY-REWARDE CFG DISTILLATION
Notations. Let ma denote an auto-regressive model parameterised by A. Given an input sequence from the space of sequences X. the model my defines policy Ty by sequential sampling of an output sequence y (81, ...,..., 81.) of length L. Specifically, given partial sequence V<n = (81, 8n-1), 1), the policy my samples the next token 8, with temperature T in the softmax such as To(S x) exp(zn/T)) where 2n ms ma(sn z) is the corresponding logit. We also define po(() II (M) the probability of sampling the output sequence y from input x.
2.1 CFG DISTILLATION FOR QUALITY
CFG. Given a model mg and a partial sequence Ven. CFG is an inference-time strategy that samples the next token by combining conditional logits mg(sn (n æ) (where æ is the user's prompt) and unconditionallnegatiee logits ma Ven, x (where x is an empty negative prompt):
Zn CCG, 7 ma mm(sm)lee ( 2) + (1 7) (s 1'<n.
CFG. The next-token probability distribution of the CFG CFG-augmented policy is then To CCC
(s,, (osllVen25 2) x
exp(2., CFG, /T). The guidance factor controls the adherence to the prompts; higher values typically lead to higher quality (Ho & Salimans, 2022). We use 7 > extrapolating rather than interpolating, as done in Kreuk et al. (2023). For example, if æ is "Soulful jazz song." and x. is "Bad audio quality.", larger 7 increases the degree which the generated sequence resembles typical elements of soulful jazz song while increasing audio quality.
CFG distillation. One key limitation that CFG doubles the inference cost, since it requires the computation of two sets of logits. To eliminate this overhead, we propose an objective that directly distills the benefits of CFG into the model weights taking inspiration from similar approaches in diffusion (Luo et al., 2023; Yin et al., 2024; Saito et al., 2024; Bai et al., 2023; Novack et al., 2024). Knowledge distillation (Hinton et al., 2015) traditionally involves compressing a large teacher model into a smaller student model (Sanh et al., 2019; Agarwal et al., 2024), allowing for efficient deployment while approximating the teacher's performance. In contrast, our teacher and student share the same architecture: the teacher (with weights 6mn from initialisation) is simply augmented with CFG. We then follow the standard distillation approach that encourages the student to match logits from the teacher.
3

---

Published as a conference paper at ICLR 2025
On-policy distillation. If the teacher provides the label, the remaining question is: how to sample the input data? i.e. the completions on which teacher's ' and student's s logits should match. Be given a set of prompts, offline distillation uses inputs sampled from the teacher (Lin et al., 2020); in contrast, we adopt the on-policy distillation strategy from Agarwal et al. (2024), where data is online and dynamically sampled from the student itself. This reduces the train-test mismatch, a.k.a the exposure bias (Bengio et al., 2015), and was used in recent state-of-the-art LLMs (Gemma Team et al., 2024)'.
CFG distillation objective. Overall, starting from the base pretrained initialisation Oinit, we distill the logits obtained by Oinic with CFG into the weights 0 without CFG by maximising:
L KL (To (-18en,
(-Iyenn x
]]
(0) = -E
Ey-no((la))
Σ
CFG, T
(2)
2.2 RL FOR DIVERSITY
Reduced diversity during alignment. Another key drawback of CFG is that it reduces the diversity of generated outputs as increases. This occurs because stronger guidance encourages the model to adhere closely to the prompt, leading to outputs that more similar each other. This trend also emerges in our CFG distillation procedure (from Section 2.1). We confirm in Figure 2 (right) that diversity across generations from the student steadily decreases as training progresses. More generally, policy collapse is recurring challenge for alignment strategies; for instance, Kirk et al. (2024) demonstrate that RLHF significantly reduces diversity of outputs. This limits exploration of novel and diverse contents, a crucial aspect for creative domains. To address this, we introduce a strategy encouraging the model to generate diverse samples. This requires two components, detailed below: (1) diversity reward and (2) a diivesity-enforriig algorithm.
Diversity reward: negative cosine similarity of embeddings. We first introduce a simple method to quantify the diversity between two generations y1 and y2. We embed those generations with model and compute their cosine similarity in the embedding space. Then, the diversity is defined as (1 82) 1 - model E. In our text-to-music application from Section 3, drawing inspiration from Futeral et al. (2024), we train a self supervised contrastive model using the semi-hard triplet loss (Schroff et al., 2015) with positive pairs coming from non-overlapping fixed-length chunks of the same audio segment. The model induces an embedding space where segments originating from the same longer audio are mapped nearby points, while segments from different recordings mapped further apart. Following Futeral et al. (2024), our embedding model takes as input a 4-seconds audio clip sampled 16kHz. The model architecture based on a compact Vision Transformer (ViT) (Dosovitskiy et al., 2021). I). More experimental details can be found in Appendix D.I.
Flys )- F((g)) The key component of this diversity reward is thus the embedding
Ely) )Il Ella]ll'
Diversity algorithm: RL for diversity. We now leverage reinforcement learning (RL) encourage the student policy to generate diverse samples for given prompt. Using the diversity reward defined above, the diversity objective to maximise is:
(0) Ex~x [E (a)pp(())))) (F Me)l]
(3)
As shown in Appendix A, an unbiased estimator of (D( is then:
(4) Due to the similarity of Equation (4) with policy gradient (Sutton et al., 1999), this diversity can be optimised with standard algorithms such as REINFORCE (Williams. 1992), updating the parameters 0 by increasing the likelihood of generations that lead to high (diversity) rewards. Overall, the only difference is that this gradient asks for multiple generations, matching the requirements of multi-sample RL algorithms recently used in RLHF (Ahmadiaa et al., 2024; Sessa et al., 2024).
(D y2) log (3(
(x) with ~ X, 31, 3/2 ~ Po (-(2)
2.3 DIVERSITY-REWARDER CFG DISTILLATION
Combining both CFG distillation from Section 2.1 and the diversity reward from Section 2.2, maximise the following novel diversity-rewardee CFG distillation objective:
QD(0) = (()) + B (0)),
(5)
Mooeovvrr, also encourage the generations from the student to optimise diversity reward, samples from the student will be readily available, making this on-policy distillation possible at no cost.
4

---

Published as a conference paper at ICLR 2025
with 3 the diversity hyperparameter scaling the RL diversity term to the KL quality distillation term. Low values of (e.g. 0) lead to policies focused on quality, while large values of 3 (e.g. 15) lead to policies prioritising diversity slight cost in terms of quality, as visible in Figure 1 (right).
2.4 MODEL MERGING FOR PARETO-OPTIMAL QUALITY-DIVERSITY
Tuning the hyperparameter 3 allows for some control over the quality-diversity trade-off before training, but exploring the full spectrum of possibilities would require maintaining a large set of models with different / values. To address this, we leverage model merging, which enables combining the strengths of different models by simply interpolating their weights (Ilharco et al., 2023; Dimitriadis et al., 2023; Ramé et al., 2023). This allows us to efficiently adjust the quality-diversity trade-off at deployment time, based on the specific needs of the user or application, with only two finetunings. Specifically, we consider two models finetuned from shared pretrained initialisation with different values of B. 8, focused on high quality (B = 0) and 0₂ focused on high diversity (high (3). We then trade-off between their abilities simply by interpolating between their weights:
OLERP (1 X) 0g- + X Oa
(6)
with À E [0, 1]. For instance, setting À close to favors the high-quality model, generating outputs that closely adhere to the prompt, while À close to 1 favors the high-diversity model, resulting in more exploratory outputs. This lightweight approach uncovers a strong and steerable quality-divg -iiveryy front of solutions with minimal computational cost, similar adjusting y in CFG during inference.
3 EXPERIMENTS
The experiments below are structured to address the following research questions. (1) CFG distillation: can distill the quality of CFG into the weights of the model, and at what cost in terms of diversity? (2) Diversity RL: can we trade-off between quality and diversity by including diversity reward during finetuning? (3) Model merging: can we construct inference time strong and steerable quality-divens front of solutions with weight interpolation?
3.1 TEXT-TO-MUSIC GENERATION: TASK, SETUP AND METRICS
Task. We explore those questions on text-to-music generation, creative task where quality and diversity are important factors. Indeed, a single music prompt could map to many different valid generations. For instance, a gentle bossa nova piece, relaxing acoustic guitar melody or an ethereal electronic soundscape would all be valid responses to the prompt "A peaceful sunset on beach". This makes -ext-io-music generation a great testbed to study quality-diversity tradc-offs.
Model and setup. We follow the experimental setup from Cideron et al. (2024), using their LLM transformer-bart bassed architecture, where a single autoregressive stage models the semantic and coarse acoustic features. We use their MusicRL- checkpoint as the base model 6mn. initialising all our experiments. We use the prompt dataset described in Section 4.1 from Cideron et al. (2024), combining multiple sources: synthetic prompts generated from the LaMDA model (Thoppilan et al., 2022), prompts collected via user interactions (Cideron et al., 2024), and prompts derived from MusicCaps (Agostinelli al., 2023). For CFG, we set n and use the negative prompt "Bad audio quality.", as it performed best on the base model (cf. Appendix B). For the diversity reward, we train a music embedding model by contrastive self-supervised learning (cf. Appendix D.1).
Training details. The partial generations for distillation are sampled from the student in an online fashion (Agarwal et al., 2024) with temperature T 0.99. The RL algorithm is variant of REINFORCE (Williams, 1992) with a baseline for variance reduction. We use a batch size of 128 and learning rate of 0. 0.00015 for all our finetunings.
Metrics. We evaluate the quality of the generations with two metrics: (1) the MuLan score (Agostinelli et al., 2023), a text adherence metric based on the MuLan embeddings (Huang et al., 2022), and (2) the User Preference score (Cideron et al., 2024), learned on 300k pairwise preferences of music generations to approximate the average human opinion score. We then define the general 5 to enforce similar range of
quality score as Squality variations for the two scores. Critically, those metrics are not used during training. In Section 3.5, we confirm the insights obtained from this quality metric through human evaluation.
8MuLan + SUser Preference using
5

---

Published as conference paper at ICLR 2025
3.2 CFG DISTILLATION FOR QUALITY
We want to validate whether or not is possible to match by distillation (at training time) the performances of CFG-augmentation (at inference time) without its inference cost overhead.
CFG distillation optimisation. Figure (left) shows the evolution the KL divergence from Equation (2) between the policy and the CFG- IFG-augmented policy along training. When the objective only to minimise the KL (i.e. 3 = 0), the KL decreases to its lowest value, implying that the CFG-free model learns to approximate the CFG logits. 3 is scheduled linearly increase until step 1000, which explains why the KI. increases after this step for / 0.
CFG distillation improves quality. Figure 2 (middle) shows that training significantly improves quality matching the performances the base model augmented with CFG and = 3. This suggests that CFG's quality can be effectively distilled into the model's weights, finding confirmed by human evaluations in Section 3.5.
CFG distillation reduces diversity. Figure 2 (right) shows that the CFG-distilled model (3 0) suffers from decrease in diversity compared to its initialisation, the CFG-free base model. Notably, after only 1k training steps, CFG-distillation reaches the same level of diversity (around 0.37) than the CFG-augmented base model (grey horizontal line for y 3). As side note, Figure 1 (right), Figure (right) and Figure 6 (in Appendix B) show that higher values y for CFG further reduce diversity. In the next subsection, we assess the efficiency of our diversity reward to promote diversity.
1600
4.5
$=15
1400
CrG
1200
Divershy
1000
4.2
B-0
8-5
10
1
4.9
0.5
3.8
400
P- 10
1000
2000
5000
3000
5000
steps
stseps
Figure 2: Left. Evolution of the KL divergence between the CFG-distilled student and the CFG- augmented teacher along training. GKD distillation alone (3 0) decreases the KL between the two policies. Middle. Evolution of the quality along training, showing improved quality for all selected values of B. Right. Evolution of the diversity across generations along training, showing that CFG distillation alone reduces diversity, but that using a diversity reward (3 of 0) can actually increase The "CFG" line shows the quality/diversity performance of the CFG-augmented base model serving as a teacher. The "upper-bound" bound" line indicates the mean diversity two generations (from the base model) for two different prompts.
3.3 RL FOR DIVERSITY
Rewarding diversity. We now analyse the results obtained when including the diversity reward from Equation (3), scaled by the hyperparameter
(0,5, 5, 10, 15) in the joint objective. As visible in Figure (right), this increases the diversity across generations along training. For = 10 or 15, diversity actually gets larger, and closer to the mean diversity between two generations for two different prompts, denoted
empirical "upper-bound".
Quality-diversin trade-off. Figure 2 (left) shows that these diversity gains come at the cost of increased KL. This is because higher values of 3 put more emphasis on the diversity term from Equation (3), hindering the minimisation of the KL distillation loss from Equation (2). As a consequence, policies finetuned with larger values of 3 have lower quality in Figure 2 (middle). Those insights are summarised in Figure (right), where the quality-diversie trade-off is plotted along training for the different values of B.

---

Published as a conference paper at ICLR 2025
3.4
MODEL MERGING FOR PARETO-OPTIMAL OPTIMAL QUALITY-DIVERSITY
0.60
LERP(O,15)
0.55
LERP(0,10)
Diversity
LERP(0,5)
LERP(O,base)
LERP(O,,510,15) uniform
0.45
CFG base model
0,40
CFG CFG-distilled model
Temperate pempeuurueeeeee base model
Temperature on CFG IFG-distilled model
0.30
3.8
4.0
4.2 Quality
4.4
4.6
Figure 3: Quality-diversity trade-off for multiple strategies. The first four lines linear interpolate (LERP) between the quality-focused model (B 0) and more diverse models (those trained with > 0, the base model), sliding À between 0 and with a step of 0.05. We also report the performance from the uniform (A })) averaging of the four models finetuned with different B, denoted "LERP(O, 5, 10, 15) uniform". We include inference-time baseline strategies CFG (when varying ?) and temperature sampling (when varying the temperature 7" - applied either on the base model or on the CFG-distilled model.
Quality-diversioe trade-off. In Figure 3 interpolate between the model with highest quality and the model with highest diversity: by sliding the coefficient À from 0 (only the quality model) to I (only the diversity model), we construct fronts of solutions that surpass the performance of individual models finetuned for intermediate B values. In particular, interpolating towards the solution with maximum diversity (B = 15) yields strong and steerable Pareto front describing the full range of possible trade-offs. This indicates that model merging achieves higher quality for given level of diversity, and vice versa. consistent with previous studies that show how merged models perform better than their non-merged counterparts (Wortsman et al., 2022a; Ramé al., 2022). This is achieved with minimal overhead: only two RL finetuning runs are required, and merging the weights involves negligible computational cost. If only one finetuning is possible, then linearly interpolating towards the base initialisation can also help, by recovering features from pretraining (Wortsman et al., 2022b); in particular note that "LERP(0, base)" outperforms the CFG-augmented base model. If more compute is available for training and we can do four finetunings, then merging them uniformly performs even better, visible Figure where the grey cross for "LERP(O,, 5, 10, 15) uniform" is (slightly) above the other fronts. This suggests that we can merge an arbitrary number of models.
CN
4.2
Qualte
2.2
LERNN 22
Figure 4: Left. Linear interpolation between the weights a model focused on quality (3 and model focused on diversity (3 11), sliding the interpolating coefficient À between 0 and 1. The dashed diagonal represents the expected values if abilities were traded-off linearly between those two models. While the diversity stays close to the diagonal, the quality remains above it, showing the benefits model merging. Right. For comparison, we also include the results for CFG when sliding y between and 7, performing worse than merged models.
7

---

Published as a conference paper at ICLR 2025
Baselines. Figure 3 also displays the results for two inference- time baselines: CFG and temperature sampling. Applied to the base model, these strategies are devenntdd by CFG-distilled models with diversity rewards. When applied to the already CFG-distilled model, they do not significantly improve quality nor diversity.
Quality-diversiie as function of X. Figure 4 (left) clarifies the effect of weight interpolation quality and diversity as we slide À between O and 1. Diversity stays close to the diagonal (representing the expected diversity) while the quality is consistently above the diagonal. This highlights the ability to significantly increase diversity with only minor quality compromises via model merging.
3.5 HUMAN EVALUATION
To conclude our experiments, we validate via human evaluation the improved quality- diversity trade-offs. We evaluate five models: the base model (we expect low quality but high diversity), the base model with CFG
Protocol.
(high quality but low diversity). the CFG-distilled model for 3 = 0 (high quality but low diversity) and 5 = 15 (medium quality and high diversity), and finally LERP(O, 15) merging uniformly with = 0.5 the two previous models (high quality and medium diversity). For quality (i.e. acoustic quality, text adherence, and musicality), we use the same evaluation protocol as in Cideron et al. (2024): the raters see two generations from different models coming from the same text prompt and rate them a scale from 1 to 5. We use 100 different prompts and each one is seen by 3 different raters. For diversity, we rely on a similar protocol except that the raters two pairs of music clips generated from the same text prompts and rate how diverse the pairs of generations are. We use 50 different prompts and each one is rated by 3 different raters.
LERP(O. 15)
LERPIO, 15)
base
B+15
GGG
base
15
CFG
..7
X
0.50
0.81
0.70
0.72
X
-05
0.50
X
0.79
0.73
0.68
Model
Model
0.19
0.21
X
0.27
LERPIO. 15)
X
LERPIO, 15)
Figure 5: Left. Side-by-side human evaluation for quality. Right. Side-by-side human evaluation for diversity. The score corresponds to the win rate of model A over model B, computed as W+L+T with W the number of wins of over B, T the number of ties, the number of losses of against This confirms that our approach improves the diversyyy trade-off. For instance, the merged model LERP(O, 15) generates music with higher diversity than the EFGG-augeetedd base model (y = 3) in 57% of the comparisons, while being rated as more qualitative half of the time (51%)
W W+T/2
Human evaluation for quality. Figure (left) presents the side-by-side win rate for quality. The CFG-distilled model (i.e. B 0) performs on par with the CFG-augmented base model (i.e. CFG) with win rate of 0.50; they both outperform the base model without CFG (first line), with 0.66 and 0.68 win rate respectively. LERP(0, 15) achieves win rate above 0.50 against all models.
Human evaluation for diversity. Figure 5 (right) presents the side-by-side win rate for diversity. The CFG-augmented base model (i.e. CFG) and the CFG-distilled model (i.e. f 0) are also evaluated similarly terms of diversity (0.52 win rate for the former); yet, they are consistently seen less diverse than the three other models. Notably, the CFG-distilled model with diversity reward (i.e. 3 = 15) has win rate of 0.73 and 0.79 against them. As a side note, this 3 = 15 model performs par with the base model (0.50 win rate), though the quantitative metrics from Figure (right) suggested that it was actually more diverse; in Section 5, we relate this discrepancy to the standard reward hacking (Amodei al., 2016 6; Gao al., 2023) phenomenon of RI.
8

---

Published as a conference paper at ICLR 2025
Human evaluation for quality-diversite trade-off. Human evaluations confirm the effectiveness of our approach in improving the quality- trade-off. The liversity-focuseee model (5 = 15) exhibits higher quality than the base model while maintaining comparable diversity. Importantly, the merged model LERP(O, 15) exhibits higher diversity than the CFG-augmented base model while maintaining comparable quality, and without incurring the additional inference cost of CFG.
Qualitative analysis. To facilitate qualitative analysis, we host music generated from all evaluated models on the website google-reseachh. github. hot.io/// ceeantt/musiclmmdivess _mrsscc An examination of generic prompts like "Rock song." confirms that CFG improves quality (i.e. less acoustic artifacts, better musicality) but reduces diversity, as generated guitars tend to be very similar across different generations. In contrast to the CFG-augmentee of rhythms while still demonstrating a clear improvement in quality over the base model. On prompts like "Opera singer" or "Male choir harmony". the quality- macuus-- model / 0 generates conventional outputs, while the diverse model | 15 produces more unconventional and creative results, sometimes leading to unexpected elements like unusual instrumentation (e.g. drums). By averaging the weights of these two models (LERP), we can effectively balance these qualities, generating high-quality music that is both faithful to the prompt and more creative.
model. the 3 = 15 model generates a wider variety
4 RELATED WORK
Classifier-free guidance (CFG). Initially introduced for diffusion models (Ho & Salimans, 2022) and later adapted for autoregressive LLMs (Gafni et al., 2022; Sanchez et al., 2023; Wings, 2022), CFG has found widespread application in various generative domains, including image (Yu et al., 2022; Saharia et al., 2022; Rombach et al., 2022; Nichol et al., 2021; Ramesh et al., 2022), video (Blattmann et al., 2023; Ho et al., 2022), and audio (Kreuk et al., 2023; Copet et al., 2023) generation. However, the detrimental impact of CFG on diversity is well-documented (Dhariwal & Nichol, 2021; Kreuk al., 2023; Nichol et al., 2021), limiting its application when exploration is key.
Knowledge distillation (Hinton et al., 2015) is emerging as a powerful technique to train state-of-the-an models (Gemma Team al., 2024). By transferring knowledge from teacher, the student can perform better than with standard training on the same data (Sanh et al., 2019; Lin al., 2020; Gu et al., 2023). In the context of diffusion models, CFG distillation was applied to drastically reduce the inference time (Meng et al., 2023; Luo et al., 2023; Yin et al., 2024; Saito al., 2024: Bai et al., 2023; Novack et al., 2024). In our work, we apply the CFG distillation idea to LLMs where we employ single-stage on-policy distillation procedure (Agarwal et al., 2024) to distill CFG-augmented I.LM, while introducing a novel model merging for improved quality-diversity trade-off
Distillation.
RI. algorithm and
Quality-diversity in LLMs. Zhang et al. (2021) compare the quality-diversity trade-offss of various inference-time strategies for LLMs, including temperature sampling, top-k sampling (Fan et al., 201 8), and nucleus sampling (Holtzman et al., 2020). These methods perform similarly except when quality is prioritized over diversity, where nucleus sampling performs best. Regarding finetuning strategies, Kirk et al. (2024); Mohammadi (2024); Chaudhari et al. (2024); Li et al. (2024) have shown that RLHF negatively impact the diversity of generated text, often measured with metrics like BLEU. To the best of our knowledge, we are the first to introduce an RI. algorithm to optimize diversity, which could potentially also solve those reductions in diversity in RLHF In contrast, existing quality- diversity algorithms (Lehman & Stanley, 201 I; Mouret & Clune, 2015; Cully et al., 2015; Cideron et al., 2020; Ding et al., 2024) aim at finding population of agents with both high-quality and diverse behaviors. Most similarly, Li et al. (2016); Zhang et al. (2018) also tried to increase the diversity across generations produced by a single agent, but measured by the number of distinet n-grams, and optimised with objectives based on mutual information (Shannon, 1948).
Model merging for Pareto-optimality. Model merging via weight averaging (Utans, 1996) has two main applications in deep learning. First, it increases generalisation by reducing variance (Wortsman et al., 2022a; Ramé et al., 2022) and memorisation (Lin et al., 2024; Ramé et al., 2024). Second, it combines their strengths (Ilharco et al., 2023; Dimitriadis et al., 2023; Wang et al., 2024), as employed in Ramé et al. (2023) for multi-objective RL, where policies finetuned with different rewards are interpolated. Similarly, we interpolate between policies where only one of them is rewarded for diversity. Despite recent theoretical efforts to explain the empirical success of model merging (Ferbach al., 2024; Ramé et al., 2024), a complete understanding remains elusive.

---

Published as a conference paper at ICLR 2025
Music generation. Diffusion models (Huang et al., 2023; Schneider et al., 2023; Liu et al., 2023) and Transformers (Agostinelli et al., 2023; Copet et al., 2023) are now the state-of-the ti-the--rt architectures for music generation. In this work, we leverage the Transformer-basee approach from Agostinelli et al. (2023), casting music generation as a categorical prediction task in the discrete token space provided by neural audio codec (Zeghidour et al., 2022; Défossez al., 2022). RI-finntunnng for music generation has previously been explored in Jaques et al. (2017); Guimaraes et al. (2017); Kotecha (2018); Latif al. (2023); Cideron al. (2024). In contrast, we are the first to distill CFG, to enforce diversity through RL, to apply model merging or to improve the
trade-off for music.
5 DISCUSSIONS AND LIMITATIONS
Amplification and distillation. In this work, we leverage a teacher-student framework where the teacher is an augmented version of the student model itself. This allows to distill the quality improvements obtained from potent but expensive inference strategy, eliminating the overhead at deployment. This echoes the principles of iterated distillation and amplification (IDA) (Cotra, 2018), where the model iteratively learns on data generated by an augmented version of itself. Examples of this can be seen in AlphaGo (Silver et al., 2016), achieving superhuman performance by distilling the knowledge of MCTS (Kocsis & Szepesvári, 2006), or in recent LLM works (Wang et al., 2023; Yu et al., 2024), distilling "System 2" into "System 1" by imitating offline predictions obtained from Chain-of- TfTuuhhh (Wei et al., 2022). Given the effectiveness of scaling inference- strategies (Snell et al., 2024), we anticipate wider adoption of such aaplification-the--iiiiit
techniques.
Extension to other models modalities. The three main components of our approach-distieee distillation of an inference strategy, RL with a diversity reward, and model merging - could be readily adapted to other generative architectures. For instance, model merging already a popular strategy for diffusion models (Purplekeyboard) 2022; Biggs et al., 2024). Additionally, while our focus is on text-to-music, similar strategies could be applied to other setups involving text, image or video generation. For applications where retaining the CFG coefficient is crucial our approach can easily be extended with methods like CLP (Wang et al., 2024). It would only require to swap the underlying RL algorithm with the CLP algorithm.
Diversity measures. We used negative cosine similarity between embeddings as our diversity measure. Critically, we demonstrate in Appendix D.2 its superior correlation with human perception of diversity compared to token-level entropy: we suspect this is because diversity should be assessed at the sentence level for creative tasks. Yet, qualitative inspection suggests that this diversity measure can still be hacked (Amodei et al., 2016; Gao et al., 2023). For example, models finetuned to maximise diversity sometimes generate excessive background drums, likely due bias of the underlying embedding model E. This could be changed by making E more invariant to background drums; alternatively, inspired by Ding al. (2024), we could learn directly human feedback (or Al-feedback) diversity embedding, by asking humans (or LLMs) to rate the similarity of generations. More broadly, relying on a single diversity metric may be insufficient, as diversity can manifest in various ways (Levy et al., 2024); in music, diversity could focus on variations in voice, key, tempo, or other elements; in NLP, a summarization task may prioritise syntactic diversity over semantic diversity. By incorporating multiple diversity rewards, each targeting a specific variation, and then leveraging model merging, we could achieve fine-grained control over diversity at deployment time.
CONCLUSION
In this work, we introduced diversity-rewarded CFG distillation, a novel finetuning approach to enhance the quality-diversity trade-off in generative models. First, we online distilled CFG, eliminat- ing its computational overhead at inference time. Second, to preserve and even enhance diversity across generations, incorporated an RL procedure that optimises diversity reward based on similarity embeddings. Third, we leveraged model merging to enable dynamic control over the quality-diversit trade-off at deployment time. Through extensive experiments text-to-music generation, we demonstrated the validity of our strategy, with our meetuned-ttenn-eette performing best according human evaluations. We believe that our work provides promising foundation for future research exploring tasks where alignment and creativity are key, and that it can be easily extended to setups beyond text-to-music generation.
model
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In ICLR, 2024. (pp. 2, 3, 4, 5, and 9)
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint, 2023. (pp. 1.3.5, 5, 10, 20, 22, and 23)
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting REINFORCE, style optimization for learning from human feedback in LLMs. arXiv preprint, 2024. (p.4)
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in Al safety. arXiv preprint, 2016. (pp. 8 and 10)
Yatong Bai, Trung Dang, Dung Tran, Kazuhito Koishida, and Somayeh Sojoudi. Accelerating diffusion-based text-to-audio generation with consistency distillation. arXiv, 2023. (pp. and 9)
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NeurIPS, 2015. (p. 4)
Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar, Yusheng Xie, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Diffusion soup: Model merging for text-to- image diffusion models. arXiv preprint, 2024. (p. 10)
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim. Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. (p.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: A language modeling approach to audio generation. TASLP, 2023. 1)
Gavin Brown, Jeremy L Wyatt, and Peter Tiño. Managing diversity in regression ensembles. JMLR, 2005. (p. 1)
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: critical analysis of reinforcement learning from human feedback for 1lms. arXiv preprint, 2024. (p. 9)
Geoffrey Cideron, Thomas Pierrot, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. Qd-rl: Efficient mixing of quality and diversity in reinforcement learning. arXiv preprint, 2020. (p.9) 9)
Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian Mc Williams, Victor Ungurcanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, Léonard Hussenot, Neil Zeghidour, and Andrea Agostinelli. MusicRL: Aligning music generation to human preferences. In ICML., 2024. (pp. 1, 3, 5, 8, and 10)
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In NewrIPS, 2023. (pp. 9, 10, and 23)
Ajeya Cotra. Iterated Distillation and Amplification, 2018 URL /Rt.tttss:/alllaaagmeeeee he//ierraedddiisee stiiiiiiiii.. latiillliiiiiiii l inntiion-anie aonn-and-aan and-ampldmmadd i f wtt.iion-55dddfffde 6. (p. 10)
Antoine Cully, Jeff Clune, Danesh Tarapore, and lean-Baptiste Mouret. Robots that adapt like animals. Nature, 2015. (p. 9)
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-tim ti-itmm dialogue. (p. 1)
11

---

Published as a conference paper at ICLR 2025
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. TMLR, 2022. (p. 10)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NewrIPS, 2021. (pp. and 9)
Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models. In ICML, 2023. (pp. 5 and 9)
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback: Towards open-ended diversity- optimization. In ICML, 2024. (pp. and 10)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition at scale. In ICLR, 2021. (pp. 4 and 20)
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP, 2023. (pp. 22 and 23)
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), ACL, 2018. (p. 9)
Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport. In AISTATS, 2024. (p.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020. (p. 2)
Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. arXiv preprint, 2017. (p. !)
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. TMLR, 2022. (p. 21)
Matthieu Futeral, Andrea Agostinelli, Marco Tagliasacchi, Neil Zeghidour, and Eugene Kharitonov.
Mad speech: Measures of acoustic diversity of speech. arXiv preprint, 2024. (pp. and 20)
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In Shai Avidan, Gabriel J.
Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), ECCV, 2022. (p. 9)
Lco Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICML,, 2023. (pp. and 10)
Google Gemini Team. Gemini: A family of highly capable multimodal models. 2023. (p. 1)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint, 2024. (pp. 4 and 9)
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation of large language models. arXiv preprint, 2023. (p. 9)
Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1331-1335. IEEE, 2024. (p.23) 23)
Gabriel Lima Guimaraes, Benjamin Sanchez-LLengeling, Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guziz Objeerive-reifforeedd generative adversarial networks (organ) for sequence generation models. arXiv preprint, 2017. (p. 10)
Sil Hamilton. Detecting mode collapse in language models via narration. arXiv preprint, 2024. (p.
12

---

Published as a conference paper at ICLR 2025
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In ICASSP, 2017. (pp. 22 and 23)
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NeurlPS, 2015. (pp. 2, 3, and 9)
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint, 2022. (pp. 1,3, and 9)
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint, 2022. (pp. and 9)
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR, 2020. (p. 9)
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. MuLan: A joint embedding of music audio and natural language. In ISMIR, 2022. (p.
Qingqing Huang. Daniel S Park, Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint, 2023. (p. 10)
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023. (pp. 2,
5, and 9)
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández- Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In ICML, 2017. (p. 10)
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In ICLR, 2024. (pp. 1, 4, and 9)
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In ECML, 2006. (pp. and 10)
Nikhil Kotecha. Bach2bach: generating music using a deep reinforcement learning approach. arXiv preprint, 2018. (p. 10)
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In ICLR, 2023. (pp. 1, 1,3, and 9)
Siddique Latif, Heriberto Cuayáhuitl, Farrukh Pervez, Fahad Shamshad, Hafiz Shehbaz Ali, and Erik Cambria. survey deep reinforcement learning for audio-based applications. Artificial Intelligence Review, 2023. (p. 10)
Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In GECCO, 2011. (pp. 2 and
Avivit Levy, B. Riva Shalom, and Michal Chalamish. A guide to similarity measures. arXiv preprint, 2024. (p. 10)
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A piversit objective function for neural conversation models. In NAACL, 2016. (p. 9)
Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling. arXiv preprint, 2024. (p. 9)
Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. arXiv preprint, 2020. (pp. 4 and
13

---

Published as a conference paper at ICLR 2025
Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of offtiiibibuttiinn generalization. In ICLR, 2024. (p.
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), ICML, 2023. (p. 10)
Simian Luo, Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv, 2023. (pp. 3 and 9)
Iariia Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong
Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv preprint arXiv:2311. 10057, 2023. (p. 23)
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR, 2023. (pp. I and
Behnam Mohammadi. Creativity has left the chat: The price of debiasing language models. arXiv preprint, 2024. (pp. and 9)
Emad Mostaque. Tweet on the examples of negative prompting, 2023. URL hwwph:tt///...tt ostaque/status stalie/sstuuuss /aeess/59999000 (0899999888899998888
(p. 18)
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint, 2015. (pp. 2 and 9)
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2021. (p. 9)
Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrice and Nicholas Bryan. Ditto-2: Distilled diffiusion inference-time t-optimization for music generation. arXiv, 2024. (pp. and 9)
Purplekeyboard. Reddit thread titled What is the point of the endless model merges?, 2022. URL https: /tt::://wwww /ww. reddit t. cww h/ttm/////b///ee// abbllieiiii.. f fus i on httmmentts/1kkaueaea waakkk/uuddwateed what t.
s.s.he.pirtt.of..teeeeddeeeeee wwwssssmoeellll mmmeerrges/// (p. 10)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint, 2022. (pp. I and
Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari. and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. In NeurIPS, 2022. (pp. 2, 7, and 9)
Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, lean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. In NeurIPS, 2023. (pp. 3, 5, and 9)
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In ICML, 2024. (p. 9)
Denise Rey and Markus Neuhauser. Wilcoxonn-sined-rrnnk test. International encyclopedia of statistical science, pp. 1658-1659, 2011. (pp. 21 and 22)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. (p.9) 9)
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NewrIPS, 2022. (p.
14

---

Published as a conference paper at ICLR 2025
Koichi Saito, Dongjun Kim, Takashi Shibuya, Chieh- Lai, Zhi Zhong, Yuhta Takida, and Yuki Mitsufuji. Soundctm: Uniting score-based and consistency models for rext-to-sound generation. arXiv,, 2024. (pp. and
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint, 2023. (p. 9)
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint, 2019. (pp. 3 and 9)
Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Schölkopf. Moûsai: Text-to-music generation with long-contex: latent diffusion. arXiv preprint, 2023. (p. 10)
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. CVPR, 2015. (pp. and 20)
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. BOND: Aligning Ilms with best-of-n distillation. arXiv preprint, 2024. (p.4)
Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In ICCV, (p. !)
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 1948. (p. 9)
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game Go with deep neural networks and tree search. Nature, 2016. (p. 10)
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint, 2024. (p. 10)
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018. (p. 17)
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. NIPS, 1999. (p.4)
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze -zee Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint, 2022. (p. 5)
Joachim Utans. Weight averaging for neural networks and local resampling schemes. In AAAI, 1996.
2 and 9)
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, et al. Conditioned language policy: A general framework for steerable multi-objective finetuning. In EMNLP, 2024. (pp. 9 10)
Peifeng Wang, Zhengyang Wang. Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. Scott: Self- consistent cfain-of-thought distillation. In ACI., 2023. (p. 10)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought prompting elicits reasoning in large language models. In NeurIPS, 2022. (p. 10)
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning, Reinforcement learning, 1992. (pp. 4, and 17)
Rivers Have Wings Tweet classifier-free guidance for autoregressive models, 2022. URL (p.
tttps://x..e com httt:ss/s...com//ineessvvvveeeee http://acaaaaaaoo........////aa.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S. Morcos. Hongseok Namkoong, Ali Farhadi. Yair Carmon. Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICMI,, 2022a. (pp. and
15

---

Published as a conference paper at ICLR 2025
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Hanna Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, 2022b. (p. 7)
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv, 2024. (pp. 3 and 9)
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022. (p.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system arXiv preprint, 2024. (p. 10)
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. Trans. Audio Speech Lang. Process., 30, 2022. (p. 10)
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina (eds.), ACL, 2021. (pp. 2, and 20)
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan, Gen- erating informative and diverse conversational responses via adversarial information maximization. NeurIPS, 2018. (p. 9)
16

---

Published as a conference paper at ICLR 2025
Diversity-Rewarde CFG Distillation
Supplementary material
A POLICY GRADIENT FOR THE DIVERSITY REWARD
In this appendix consider the general case of optimising diversity across N generations, where:
rp( Ei.je( (D(M,,
EiJe(L...N).as))
The key originality is that this reward considers multiple trajectories simultancously, while the traditional policy gradient from Sutton (2018) considers single trajectory. We make the theoretical connection below.
Let's consider symmetric multi-trajecttrres. reward, where the expected reward is defined follows:
(
N En-x [Em.... ( (y .... ....N]]]]]
(8)
If we expand the inner expectation:
By. uy (po(-ps) [FD(y),.., UN]] Σ
rD yN II po(u)12).
MN
Then take the gradient:
VE Big...... uw (rD( uN)] Vo
Σ TD .... UN) II po(uilz)
(10)
N II po(v))
Σ
(po(usle)rDlo UN)
(11)
B1 MN j=1
1
Σ
Il (
V log (2(
rD(uI, ....)))
(12)
,I 8ml
Vlogpo(ylzE)rD( yNd
II pe (v(lz)
(13)
1=1 11....
N
N Σ Vlogro(y)lz)rD(en yNd (o())))).
(14)
BI....yN
For Equation (12), used Vlogr Vu/x. For Equation (14), we used the symmetry of and change of variable. Hence, non biased estimator of the policy gradient for ( is:
TD(U .... HN V log (o(()))))) where
X. 91. .... UN ~ (-(=)))
(15)
Due to the similarity of Equation (15) with the estimator of policy gradient for single-trajectory rewards, the diversity reward can be optimised with the usual policy gradient algorithms. As an exam- ple, the sampled gradient with a batch size B of the vanilla REINFORCE algorithm (Williams, 1992) would become
BN EB
B Ej rD (Vi,1, 1 ... Vi.N) V log po (Vi,g (1:) with 2; X, Mk. po (-(-
17

---

Published as a conference paper at ICLR 2025
B CFG FOR TEXT-TO-MUSIC GENERATION
Figure shows the impact of CFG when applied at inference time on the MusicRI-R base model. Specifically, we plot the quality-divesiity front of solutions when sliding the adherence hyperparame- ter for unconditional and negative prompting. We observe that CFG significantly decreases diversity going from 0.5 (for y 1) 0.3 (for y 7). Critically, we also see that the front of solutions for negative prompting with "Bad audio quality." is above the one revealed by unconditional prompting. This specific negative prompt, inspired by those used image generation (Mostaque, 2023), was selected based on preliminary experiments where it outperformed other negative prompt candidates. Finally, for quality the best y 3, which we use throughout this work.
negative prompting CFG unconditional CFG
0.50
0.45
V= 2
Diversity
Y=
0.40
Y
Y
0.35
K
Y
7
0.30
V=5
3.8
4.0
4.2
4.6
Quality
Figure 6: Effect of CFG on quality and diversity as function of The quality is greatly improved the expense of diversity. Negative prompting outperforms unconditional prompting.
Figure shows the effect of CFG the User Preference score and the MuLan score. Unconditional CFG improves the User Preference score and the MuLan score which indicates that the generations both of better quality and of better adherence the text. For negative prompting CFG, the User Preference score gain is doubled compared to unconditional. However, the MuLan score decreases which can be explained as the generations are biased towards good acoustic quality. With negative prompting CFG, the model tends to always produce high quality audio even when the prompt does not explicitly mention it.
18

---

Published as a conference paper at ICLR 2025
0.6
0.84
User Preference
0.4
0.2
MuLan
0.81
0.0
0.80
-02
0.79
negative prompting CFG
negalive prompting CFG
-0.4
unconditional
V
Figure 7: Effect of CFG on the User Preference score and the MuLan score. Negative prompting CFG improves twice much the User Preference score. For unconditional CFG, the MuLan score which reflects the text adherence increases which is an expect effect of CFG. For negative prompting CFG, there drop of MuLan which can be explained as we biased the generations towards good audio quality' which is something that might not be present in the text prompt.
C DETAILS ON THE EVALUATION METRICS
Figure 8 shows the User Preference and the Mul.an score for the different models. Figure 8 shows that (1) = matches the performances CFG on both metrics and (2) increasing the coefficient primarily impacts the text adherence score while keeping the User Preference score nearly constant. It shows that that our approach improves the diversity by reducing the text adherence without impacting the audio quality.
3,550
0.-
0.750
0.0
3.725
3.700
3.575
3.50
bage
CFG
B= 0
5
10
1=
base
CFG
B 15
Figure Comparison of the performances of the models on the User Preference score and MuLan score. B 0 matches the performances of CFG. Increasing the 3 coefficient primarily impacts the text adherence score while keeping the User Preference score nearly constant. It shows that our approach improves the diversity by reducing the text adherence without impacting the audio quality.
19

---

Published as a conference paper at ICLR 2025
Entropy per token
Human
#515
ures Ar1
swp
useio A33
8-
Lewio 251
Figure 9: Comparison between different measures of diversity; the one provided by our embedding model (left), the Elo rankings provided by human evaluation of diversity (middle) and the entropy per token (right). CFG inference time (yellow) reduces all those measures of diversity. Models trained with our RL diversity reward (3 = 15 green) have increased diversity according to our embedding model and human evaluation, but not in terms of entropy.
D DIVERSITY IN MUSIC
D.1 EMBEDDING MODEL FOR DIVERSITY
Diversity in music. The objective our diversity reward is to estimate the dissimilarity between two music segments. Diversity, especially in music, is multifaceted concept encompassing various aspects such as instrumentals, melodies, tempo. In this work we conceptualise diversity through the following question: Are these audio excerpts derived from the
audio recording not?
Embedding. Drawing inspiration from Futeral et al. (2024), we train a self supervised contrastive model with positive pairs coming from non-overlapping fixed-length chunks of the segment. The model induces an embedding space where segments originating from the same longer audio are mapped to nearby points, while segments from different recordings are mapped further apart. This is useful because cosine similarity in the embedding space can quantify the similarity between two audio segments.
audio
Details. Following Futeral et al. (2024), our embedding model takes as input 4-seconds audio clip sampled 16kHz. We generate the -pe-sppetroggam using window size of 512 samples, hop length of 256 samples, and 156 frequency bins. This results in 376 time frames, each with 156 dimensions. The model architecture founded on a compact Vision Transformer (ViT) (Dosovitskiy et al., 2021 comprising 12 layers. Each layer incorporates 6 attention heads, an embedding dimension of 512, and a feed-for foed-forward layer with a dimension 1024 (totaling 25 million parameters). The final embeddings averaged over time and then projected into 9i-eennnsio space. For training, semi-hard triplet loss (Schroff et al., 2015) with a total batch size of 3840, for 200,000 steps, using 16 kHz audio excerpts sourced from the same training dataset Agostinelli et al. (2023). Additionally, we augment the training data with noise, tempo and pitch shifts.
employ the
D.2 COMPARISON OF OUR DIVERSITY METRIC WITH HUMAN EVALUATION AND ENTROPY
We compare the corrllation between our diversity metric, the Elo score computed from human diversity evaluation in Section 3.5, and the entropy per generated token (which is commonly used proxy for diversity (Zhang et al., 2021)). Figure highlights that our diversity measure has higher correlation with the human raters' notion diversity than entropy. As an example, / = 15 scores high both in terms of Elo score and diversity metric, while the entropy per token is low as CFG (which is low diversity approach). Yet, our measure of diversity remains imperfect and thus can be hacked, further discussed in Section 5.
20

---

Published as a conference paper at ICLR 2025
D.3 OPTIMISATION OF THE DIVERSITY REWARD ACCORDING THE NUMBER OF GENERATIONS.
To compute the diversity reward ,.. yN), one important parameter is N the number of generations. Figure 10 shows the quality (left) and diversity (right) obtained when varying the number of generations N (2,4, 8) while keeping constant the batch size 128. Figure 10 shows that N = is the best hyperparameter to optimise the diversity. Hence, we used N 2 throughout our experiments.
4.5
0.675
num gen
gen-4
0.650
gen=a
4.3
4.2
Diversity
Quality
0.600
0.575
3.9
0.550
num gen=2
3.8
num gen-4
0.525
num gen-8
0.500
0
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
# steps
# steps
Figure 10: Comparison between different values of N the number of generations per prompt. In this experiment we ran the same experimental setup (3 = 15, batch size= 128) with N 2, 4, The results show that while the quality score is roughly the same for every values N (left), the diversity score is best optimised with N = 2 (right).
D.4 VENDI SCORE VS MEAN SIMILARITY
For the 101 prompts used for the quality human evaluation, we generated 8 music clips and we computed the diversity of these music clips with two measures (1) simple 1-mean computed on the similarity matrix, and (2) the Vendi score (Friedman & Dieng, 2022) computed on the similarity matrix. Table shows that both measures achieve the same ranking in term of diversity.
Table I: Comparison of the diversity score computed on generations generated with the prompts from the human evaluation. The diversity score is either computed with simple 1-mean of the similarity matrix or with the Vendi Score (Friedman & Dieng, 2022) computed on the similarity matrix.
Model
Vendi Score
Mean
8 = 15
2.91
0.60
base
2.40
0.46
LERP(O, 15)
2.35
0.44
3=
1.95
0.32
CFG
1.93
0.32
E DETAILS FOR HUMAN EVALUATION
Table 2 and Table show the details results of the side by side human evaluation for respectively quality and diversity. The p-values for statistical tests, a paired t-test and the Wilcoxon signed rank test (Rey & Neuhäuser, 2011) are displayed in these tables. Our results are statistically significant according to both statistical tests.
21

---

Published as a conference paper at ICLR 2025
Table Details of the side-by-side quality human evaluation. Two statistical tests were performed on the results: the paired t-test and the Wilcoxon signed-rank test (Rey & Neuhäuser, 2011). The p-values for these two tests are displayed in the last two columns.
Model A
Model B
#WINSA
#WINSB
#DRAWS
paired t-test
Wilcoxon test
base
CFG
59
168
76
6.5e-14
1.0c-12 12
base
B = 0
62
161
80
6,3e-13
5.0e-12 12
base
B 15
82
130
91
2.8e-04
3.0e-04
base
LERP(O, 15)
49
184
70
1.7e-22
1.4e-19
CFG
B 0
107
104
92
7.4e-01
7.3e-01
CFG
β = 15
133
78
92
2.7e-05
3.1e-05
CFG
LERP(O, 15)
96
105
102
8.0e-01
8.2c-01
8 = 0
B = 15
123
95
85
1.9c-02
1.8c-02
3 LERP(O, 15)
99
109
5.1e-01
5.1e-01
8 = 15
LERP(O, 15)
74
140
89
9.0e-06
1.4e-05
Table 3: Details of the side-by-side diversity human evaluation. Two statistical tests were performed on the results: paired I-test and the Wilcoxon signed-rank test (Rey & Neuhäuser, 201 The p-values for these two tests are displayed in the last two columns.
Model
Model B
#WINSO
#WINS1
#DRAWS
paired t-test
Wilcoxon test
base
CFG
107
23
20
9.6e-14
1.8e- 11
base
114
21
15
9.4e-20
1.0e-15
base
B = 15
56
57
37
6.2e-01
6.4e-01
base
LERP(O, 15)
95
28
27
1.5e-08
2.4e-07
CFG
8 = 0
62
56
32
5.8e-01
5.3e-01
CFG
3 15
30
100
20
2.1e-11
3.6e-10
CFG
LERP(O, 15)
53
74
23
2.3e-02
1.6e-02
B 15
22
108
20
2.6e-16
1.0c-13
β 0
LERP(O, 15))
51
75
24
3.9e-02
5.3e-02
=
LERP(O, 15)
92
37
21
2.6e-07
4.9e-07
F CLAP SCORE METRICS
For the 101 prompts used for the quality human evaluation, we generated music clips and computed the CLAP score (Elizalde et al., 2023), As we generated 8 music clips per prompts. we computed mean CLAP score and a standard deviation. Table 4 shows the CLAP score for the different models. All models perform equally.
Table 4: Clap scores computed on the 101 prompts used for the quality human evaluation. All models perform equally.
Model
Clap Score 0.52 + 0.11
Beta=0
Beta=15
0.52 + 0.11
LERP(0, 15)
0.52 0.11
Base
0.53 + 0.11
CFG
0.53 + 0.11
G FAD SCORE
Table shows the FAD score computed on MusicCaps (Agostinelli et al., 2023) with two audio embedding backbones namely CLAP (Elizalde et al., 2023) and VGGish (Hershey al,, 2017) and
22

---

Published as a conference paper at ICLR 2025
the FAD score computed on SongDescriber (Manco et al., 2023). We see similar pattern for all the variations. Without diversity (i.e. 3 = 0) CFG Distillation matches the performance CFG. Then, increase diversity, FAD decreases consistently. However, there is an inconsistency the base model has lower FAD than CFG which is contradicted with the quality human evaluation that shows that CFG is better than base. The inconsistency the FAD score reported in previous work (Copet et al., 2023; Gui et al., 2024).
Table 5: FAD on MusicCaps (Agostinelli et al., 2023) computed with either CLAP (Elizalde et al., 2023) or VGGish (Hershey et al., 2017) and FAD with CLAP SongDescriber (Manco et al., 2023). We see similar pattern for all the variations. Without diversity (i.e. 3 = 0) CFG Distillation matches the performance of CFG. Then, as we increase diversity, FAD decreases consistently. However, there is an inconsistency as the base model has a lower FAD than CFG which is contradicted with the quality human evaluation that shows that CFG is better than base.
Dataset
MusicCaps
MusicCaps
SongDescriber FAD w/ CLAP
Audio Embedding FAD w/ VGGish
FAD w/ CLAP
base CFG B == 0
3.759
0.15
0.11
4.371
0,21
0.13
4.357
0.21
0.13
LERP(0, 15)
4.986
0.23
0.13
B = 15
6.437
0.26
0.15
23


Reviews:
[{"review_id": "2eVdowZJtj", "reviewer": "Reviewer_EZJJ", "summary": "This paper introduces a fine-tuning method that enhances both quality and diversity in generative models, targeting CFG’s limitations in computational cost and diversity reduction. The authors propose a two-part training objective: distilling CFG’s quality-enhancing effects into model weights and using reinforcement learning to encourage diversity. They also introduce model merging, which allows flexibility in balancing quality and diversity at deployment. Experiments on MusicLM show improvements over CFG, with human evaluations confirming superior quality-diversity trade-offs.", "questions": "About music embedding\nWhy not use the open-source music embedding models to facilitate the reproducibility of this work? \n\nAre there any results of embedding evaluation like acc of linear probing that can be public for the music embedding model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper presents an innovative approach to overcoming CFG’s limitations, particularly by combining distillation with a diversity-promoting reinforcement learning reward. It proposes a metric of diversity based on pair-level dissimilarity. \nQuality: The methodology is robust, with well-designed experiments that rigorously evaluate quality-diversity trade-offs. The use of human evaluations adds significant credibility, and the proposed model-merging strategy for adjustable quality-diversity control demonstrates strong, reliable results.\nClarity: The paper clearly articulates its objectives, methods, and findings, making complex topics accessible. Detailed diagrams and well-structured explanations approach and results easy to understand, enhancing the overall readability and impact.\nSignificance : By improving both quality and diversity without increasing inference cost, this work addresses an important limitation in generative models. While primarily applied to music generation, the approach has potential implications across other generative domains, offering a valuable contribution to fields where creativity and efficiency are essential.", "weaknesses": "About the diversity training\nAccording to equation 3, D(\\theta) dependents on the expected value of dissimilarity of y given x. How many ‘y’s  are sampled given an x ? Besides, this paper does not further investigate the impact of the sample number of an x on the diversity training . \n\n\nThe definition of diversity is based on pair-level dissimilarity. It may be better to use some metrics like gini or Kurtosis to measure the global diversity of the generated songs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CKuTQHJxx", "reviewer": "Reviewer_G1MN", "summary": "The paper proposes a method called Diversity-Rewarded Classifier-Free Guidance (CFG) Distillation for improving generative models, particularly in creative tasks like music generation. The authors introduces two training objectives:\n1. Distillation: The method distills CFG’s quality improvements directly into the model weights, which eliminates the extra computational overhead during inference.\n2.Reinforcement Learning (RL) for Diversity: A reinforcement learning objective rewards the model for producing diverse outputs, balancing the quality-diversity trade-off.\nThe approach focuses on a text-to-music generation task, where it successfully improves quality and diversity, validated by human evaluations.", "questions": "1. Rewarding to Weakness 2, would you mind including FAD or other metrics to state the quality? \n2. \"Diversity\" is conceptualized as \"Are these two audio excerpts derived from the same audio recording or not?\". While this diversity is essential to study and learn, since we are dealing with music, is that possible that we can have better conceptualization of \"Diversity\"? For example, \"Given one reference audio recording, how diverge are these two audio excerpts in terms of genre comparing to the reference audio?\" In short, I think the word \"Diversity\" can be more music related.\n3. Same logic applies to \"Quality\". Audio quality can be applied to speech as well. Since this work only focuses on music, can \"Quality\" be more music focused? For example, the structure and key coherence in the short excerpt. \n4. Again, RL with a diversity reward objective seems like a general objective used or inspired from other fields, like population-based RL. While the application to music is novel, I don't think this is novel enough to be highlighted as novelty. Would you mind defending your argument of novelty? Why this objective special to music (this objective can be applied to speech as well)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This work provides a new strategy for managing the quality-diversity balance in generative models and has potential applications in other areas of creative content generation.\n2. To achieve both diversity and quality, instead of doing direct multi-objective training which can downgrade the performance, the authors first distill model in terms of audio quality with classifier free guidance, which increase the quality while maintaining the capability of text understanding to music.\n3. The analysis of quality-diversity tradeoff provides further insights to music generation models, where quality and diversity are actually two factors that need to be balanced.\n4. Great demos and great results! Did not listen to all of them, in particular, I like LERP 50/50 of “Melodic Danceable Brazilian Music with Percussions” the most.", "weaknesses": "1. For music, the terms “quality” and “diversity” are not obvious in the paper. “Quality” can not only refer to audio quality, but can also refer to other aspects of music, like style consistency, the key consistency, rhythmic patterns, etc. I assume “quality” in this paper means the audio quality only. \"Diversity\" is specified in Appendix C.1, which I think it \"Quality\" and \"Diversity\" should be mentioned clearly in the introduction or methodology section.\n2. In terms of evaluation of music, using MuLan score is great in terms of text coherence of generated music, while assuming “quality” means audio quality, it would be nice to include Fréchet Audio Distance (FAD) (Kilgour et al., 2019).\n3. RL with a diversity reward objective maybe is newly applied to the realm of music, but this is not new in the RL world. I believe at least the work “Effective Diversity in Population Based Reinforcement Learning” (Parker-Holder et al., 2020) is worth mentioning and I did not see this work is cited in the current paper.  I would like to see some editing in the paper to change the wordings. If I miss this citation or relevant citations, please point out and correct me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uGI0JPWSnr", "reviewer": "Reviewer_5uP1", "summary": "In this work, the authors introduce a novel finetuning recipe for increasing the runtime efficiency of the model and improving diversity at the same time, through Diversity-rewarded CFG distillation and model merging. The authors show that by distilling the CFG behavior into the model with an RL-based diversity regularizer and then merging high diversity with high-quality model finetunes, they can achieve strong diversity and quality tradeoffs and improved results over the base model.", "questions": "- Line (254-255): In discussing batch size, is this for the number of samples or the number of pairs samples generated? In general, it is not clear when discussing the RL objective how these pairs are chosen in the context of training, and whether the RL objective is only calculated on single pairs of samples or multiple pairwise comparisons.\n- In Figure 3, is there a reason there is only a single LERP data point for the 4-way merge? It would be useful to see if further interpolation between the 4 models is able to achieve a strictly dominant Pareto front above the LERP(0,15) model, as the claim that “This validates that we can merge an arbitrary number of models.” (Line 359) seems somewhat under-validated in this context.\n- Line 962, what is 1-similarity?\n- While certainly not important, I wonder if the authors have insight as to how techniques from LLMs in the contrastive decoding literature (especially works like DoLa which are self-contrastive) are related to CFG for autoregressive LM TTM models. In fact, the update equations from CFG and contrastive decoding/DoLa are practically identical, as they both involve the log difference in probabilities between the base model and some “worse” reference model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposed CFG distillation method seems strong and is reasonably novel in the context of autoregressive LM models\n- As a whole, proposed method is exceedingly clear, and while simple, shows the strengths of each facet of their 3-pronged method articulately.\n- The significant time dedicated to discussing results and ablations is particular welcome, and the authors do a great job of exploring the design space of their method.\n- The insight from Fig. 4 is in particular quite fascinating, do you have any idea why this merging behavior actually improves quality beyond the interpolating models?", "weaknesses": "Overall, while I think this is a reasonably strong work, there are a number of concerns I have addressed below that shift my overall recommendation to reject. In fixing some or all of these issues (in particular improving the discussion of CFG distillation novelty, using publicly available eval metrics such as FAD/CLAP/Recall/Coverage, including human eval error bars / statistical tests, and focusing more on the diversity model and its calibration in the main text) I would raise my score accordingly.\n\n# Overclaiming of CFG Distillation Novelty:\nIn general, while the core contribution of the CFG distillation approach is interesting, I think it’s stated novelty is an over claim (such as the line 492-493 “In contrast, we are the first to distill CFG…for music,” which is false). While CFG Distillation may be novel in the case of autoregressive audio generation (MusicLM, MusicGen), CFG distillation is a common practice in diffusion models for the same reasons motivated in this paper (namely, reducing inference cost), normally coupled with other distillation techniques. While the authors rightly mention Meng et al., this work is rather old and the stated novelty of their “two-stage” approach vs. the authors’ “single-stage” approach (461-462) does not really hold, as most works after Meng et al. (and before the present work) use a single stage approach (and in general, the paper is presented as if the whole of CFG distillation is a fully novel contribution). Single-stage CFG distillation has been a standard practice in image domain distillation [1, 2], but more saliently has appeared in a number of audio (including music) generation [3,4] and music-specific generation [5] distillation works. As an actionable change, it would be useful to explicitly mention all of these connections in the paper (in both the introduction, the related works, and the section on CFG distillation), and make clear that the novelty of the proposed method comes from how they achieve CFG distillation (and the first to do so on autoregressive LM-based music models), rather than the *idea* of CFG distillation for music generation.\n\n\n# Reduced Discussion on Diversity Model:\n- It is odd that perhaps the most novel portion of the paper of the RL Diversity Objective (as both CFG distillation and model merging are standard practices) is relegated to most details of the core embedding model being in the appendix. To me, this is a core part of the paper (as defining diversity through some model underpins the entire method, and is very interesting), and it would be useful to include this in the main body of work (and I think there is a good deal of text in the results section that is redundant and could be condensed). \n\n# Issues with Evaluation Suite:\n- While the authors make no mention of open sourcing model weights or code (which in my opinion is fine and shouldn’t be counted as a negative against them), the overall evaluation suite **also** uses fully closed-source models, severely limiting the reproducibility of the work and how further research could use it as a baseline. While I am sure that the MuLan embeddings and User Preference score are fine metrics in their own right, they are wholly inaccessible to the wider community. To accurately assess how the proposed method performs, not only to their own baselines but also external models, it is imperative to include metrics that are publicly usable in the literature, such as CLAP score to assess text-relevance, FAD/MMD (with CLAP backbone) to assess audio quality w.r.t. a reference set (like SongDescriber), and so on. I very much recommend the authors include some number of these (such as FAD and CLAP) in order to contextualize their performance in the larger space of TTM generation. In parallel, running existing open source models (such as AudioLDM2 or Stable Audio Open) on the present closed source eval suite would also work.\n- The definition of “quality score” (259-260) is somewhat divorced from the literature on TTM and generative evaluation. This seems to combine MuLan score (theoretically a measure of “text relevance”) and user preference, but this explicitly entangles these two different axes of behavior that are generally kept separate (as how previous works do not combine CLAP and FAD for instance [3,4]). It should either be explicitly stated as to *why* this combination can be made, or more explicitly simply uncombine them and present all results for user preference and MuLan score separately, as currently it is impossible to tell *which* of these metrics the proposed method actually improves.\n- I have strong worries on the overall statistical power of the human eval. As the human eval is a core part of the presented results, both in order to prove the efficacy of the proposed method and to back-verify that the diversity model correlates with human perception, it is unclear what sort of statistical significance actually occurs here, as there seem to be no hypothesis tests run nor error bars reported for any of the human eval metrics, exacerbated by the fact that there seems to be only 3 users in the listening study (which is far lower than even the small sample sizes common in generative media research). Specifically, it would be useful to see paired t-tests run between each method on the scores (averaged across human raters) to determine significance.\n- There is some circularity to the evaluation of diversity in this work, despite it being a core focus. Though the authors rightly identify that the quality score is a good measure as it is not used in training, using the diversity model for eval and RL fine-tuning by nature biases results towards the proposed method (as it directly optimizes for this result). This ignores the fact that there are a number of established different metrics for assessing diversity in generative media, including Inception Score (IS), Recall, Coverage, and Vendi Score to name a few. Including some or any one of these scores would be useful both to bring it more inline with existing TTM literature as well as assess the output diversity of the model in an unbiased fashion and help verify whether the trained diversity model correlates with these values.\n- Along with the above 2 points, the overall justification of the diversity model’s success hinges on Figure 7. (Middle)’s analysis that the fine-tuned model shows similar Elo ranking to the base model. I’m not sure if Elo score is the best measure for this, as it is a purely relative measure and details on its exact calculation are not provided.  It would be better to show the *absolute* scores for each model (with error bars) and see how well this correlates with the diversity model, and in general justify more in the paper that the diversity model is reasonably calibrated given its underpinning for the whole work.\n\n[1] Luo, Simian et al. “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference.” ArXiv abs/2310.04378 (2023): n. pag.\n\n[2] Yin, Tianwei et al. “Improved Distribution Matching Distillation for Fast Image Synthesis.” ArXiv abs/2405.14867 (2024): n. pag.\n\n[3] Bai, Yatong et al. “ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation.” Interspeech 2024 (2023): n. pag.\n\n[4] Saito, Koichi et al. “SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound Generation.” ArXiv abs/2405.18503 (2024): n. pag.\n\n[5] Novack, Zachary et al. “DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation.” ArXiv abs/2405.20289 (2024): n. pag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DiSBcDSQGb", "reviewer": "Reviewer_suju", "summary": "The paper proposes a distillation based method to improve text to music models (specifically MusicLM and MusicRL). The student network learns the CFG-modified logits of the teacher model, and therefore running the teacher model with CFG is approximated with running the student network directly, which reduces inference compute to half. To address less diversity caused by CFG and by the CFG distillation, the paper proposes to maximize the diversity computed from inter-similarities during CFG distillation. The paper also finds directly interpolating two model weights computed from different $\\beta$ leads to better quality-diversity tradeoff.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper is extremely well written and the motivation is very clear. It aims to solve two very important challenges in text-to-music generation: inference overhead and bad quality-diversity tradeoff. The proposed method makes a concrete understanding towards these challenges.\n\nThe methods including CFG distillation, diversity maximization, and the use of model weight interpolation are straighforward and simple enough that these ideas could be potentially easily and widely adapted to the deep generative model community. \n\nThe experimental results show concrete improvements on the quality-diversity tradeoff compared to the base model with or without CFG.", "weaknesses": "The major weakness of the CFG distillation is that it destroys the student model's ability to take different CFG scale as input, which is key in many downstream applications. I understand that this could be useful in applications where the inference strategy (i.e. CFG) is fixed by nature, but in many cases we need the flexibility to take different CFG scales at inference time. Therefore, I would expect the student model to take the CFG $\\gamma$ as inputs as well, which could be done in a similar way how diffusion models are conditioned on timesteps.\n\nThe second weakness is that in Figure 5, the proposed model does not outperform CFG significantly. The quality is about the same, and the diversity is 57-vs-43. In addition, the diversity is much worse than the base model after model weight merging, indicating that the merged model still suffers from lack of diversity. \n\nThe third weakness is that the paper does not evaluate the proposed distillation method on other baseline models and tasks (e.g. audio generation and speech generation) -- especially open-sourced models -- which limits the contribution of the paper. As a result, it is unclear how general the proposed method is when applied in other data and tasks. \n\nMinor weaknesses:\n\nOne minor weakness is that the paper does not analyze in depth the diversity drop of the vanilla CFG distillation. From Fig 2 one could infer it is most likely because the student model learns the CFG-modified logits well, but it is also possible that the KL loss in eq (2) amplifies it. It would be useful to study different divergence losses and different model capacities.\n\nAnother minor weakness is that Fig 3 is almost impossible to read. Please improve the display of this figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)