Diversity-Rewarded CFG Distillation

## Abstract

Abstract Generative models are transforming creative domains such as music generation, with inference-time strategies like Classifier-Free Guidance (CFG) playing a crucial role.
However, CFG doubles inference cost while limiting originality and diversity across generated contents.
In this paper, we introduce diversity-rewarded CFG distillation, a novel finetuning procedure that distills the strengths of CFG while addressing its limitations.
Our approach optimises two training objectives:
(1) a distillation objective, encouraging the model alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL objective with a diversity reward, promoting the generation of diverse outputs for a given prompt.
By finetuning, we learn model weights with the ability to generate high-quality and diverse outputs, without any inference overhead.
This also unlocks the potential of weight-based model merging strategies: by interpolating between the weights of two models (the first focusing on quality, the second on diversity), we can control the quality-diversity trade-off at deployment time, and even further boost performance.
We conduct extensive experiments on the MusicLM (Agostinelli et al., 2023 ) text-to-music generative model, where our approach surpasses CFG in terms of quality-diversity Pareto optimality.
According to human evaluators, our finetuned-then-merged model generates samples with higher quality-diversity than the base model augmented with CFG.
Explore our generations at google-research.github.io/seanet/musiclm/diverse_music .

###### keywords:

## 1 Introduction

Generative models for creative domains.
Art and entertainment — domains historically driven by human creativity — are undergoing a profound transformation thanks to AI generative models.
These models, often powered by Large Language Models (LLMs) or diffusion, can now generate texts (Gemini Team, ), images (Ramesh et al., ), videos (Ho et al., ), and audios (Borsos et al., ; Kreuk et al., ; Copet et al., ; Agostinelli et al., ; Cideron et al., ; ).
To further refine the quality, these models are often augmented with inference methods during real-world deployment, ranging from simple temperature scaling to more refined methods like Beam search (Freitag and Al-Onaizan, ), test-time augmentation (Shanmugam et al., ), or MCTS (Kocsis and Szepesvári, ).
A particularly popular method for image and audio generation is classifier-free guidance (CFG) (Ho and Salimans, ), e.g., used in DALL-E (Ramesh et al., ) or AudioGen (Kreuk et al., ).
CFG improves the model’s fidelity to the prompt by combining the logits of conditional and unconditional generations.
Despite its benefits, CFG has two main limitations: it doubles the computational cost during deployment and reduces the diversity of generated content (Ho and Salimans, ; Dhariwal and Nichol, ; Kreuk et al., ; Meng et al., ), hindering the exploration of novel and diverse ideas — a cornerstone of creativity.
Ideally, these models should not only fulfill user intent but also surprise them with unexpected and innovative outputs: the model should not generate systematically the same content (Hamilton, ).

Quality-diversity trade-off.
Effectively controlling the quality-diversity trade-off is thus extremely important but challenging.
On the one hand, optimising quality usually reduces diversity; this limitation affects inference methods such as CFG but also finetuning stages such as RLHF (Kirk et al., ; Mohammadi, ).
On the other hand, promoting diversity usually reduces quality (Brown et al., ), e.g., when increasing temperature at inference (Zhang et al., ).
As further discussed in Section 4, quality-diversity algorithms (Lehman and Stanley, ; Mouret and Clune, ) seek to train a population of models with diverse abilities.
In contrast, optimising directly the diversity of generations of a single model is an under-explored yet promising avenue, that we investigate here.

Diversity-rewarded CFG distillation.
In this work, we introduce a novel finetuning strategy to enhance the quality-diversity trade-off in generative models for creative domains. Specifically, we combine distillation and reinforcement learning (RL) to optimise two complementary objectives.
The first is a novel *CFG distillation objective* where we distill the behavior of a teacher into the student.
Critically, the teacher is the CFG-augmented base model, rather than a larger third-party model as commonly done in the literature (Hinton et al., ).
This involves minimizing the KL divergence between the logits of the teacher and the student on the data distribution generated by the student (to reduce train-test mismatch), following the on-policy distillation framework of Agarwal et al. ().
By distilling CFG into the model weights, we improve generation quality while eliminating CFG’s inference overhead.
The second is a novel *RL with a diversity reward* objective, maximising the diversity across pairs of generations for a given prompt.
Diversity is measured with by comparing pairs of generations by first embedding them and then computing their negative (cosine) similarity in this embedding space.
Combining these two objectives allows the finetuned model to inherit the quality of CFG without its cost, while simultaneously maintaining diversity through the RL objective, thus solving the two main drawbacks of CFG at inference time.

Model merging.
The hyperparameter $\beta$ (multiplier for the diversity reward) allows controlling the quality-diversity trade-off *at training time*, as shown in Figure 1 (right); in contrast, traditional CFG can do it *at deployment time*.
To enable this level of control within our approach, we propose a third contribution involving model merging. Specifically, we finetune two models, one focusing on quality (low $\beta$) and the other focusing on diversity (high $\beta$), and then combine their weights by *linear interpolation (LERP)* (Utans, ), balancing between quality and diversity.
This follows from the linear mode connectivity property (Frankle et al., ) and recent advances in model merging (Wortsman et al., ; Ramé et al., ; Ilharco et al., ), showing that weights finetuned from a shared pretrained initialisation can be interpolated, despite the non-linearities in the architecture.
Notably, Ramé et al. () showed that interpolating between weights finetuned on different rewards trades their abilities off.
Thus, we interpolate between the weights of a quality-focused model and a diversity-focused model: sliding the interpolating coefficient $\lambda$ between 0 0 and $1$ uncovers a strong and steerable quality-diversity front of solutions, without overhead at deployment.
Crucially, we find that the interpolated model using $\lambda=0.5$ is our best model, outperforming models explicitly finetuned for intermediate values of the diversity hyperparameter $\beta$.

Music generation.
We apply our strategy to text-to-music generation, a creative task where balancing quality and diversity is key.
Specifically, we finetune MusicLM (Agostinelli et al., ) and consistently improve the quality-diversity trade-off achieved by the CFG-augmented previous state-of-the-art (Cideron et al., ).
Our experiments, featuring human evaluations, validate that our models generate more diverse music while maintaining high quality.

## 2 Diversity-rewarded CFG distillation

Notations. Let $m_{\theta}$ denote an auto-regressive model parameterised by $\theta$.
Given an input sequence $x$ from the space of sequences $X$, the model $m_{\theta}$ defines a policy $\pi_{\theta}$ by sequential sampling of an output sequence $y=(s_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},s_{L})$ of length $L$.
Specifically, given a partial sequence $y_{<n}=(s_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},s_{n-1})$, the policy $\pi_{\theta}$ samples the next token $s_{n}$ with temperature $T$ in the softmax such as $\pi_{\theta}(s_{n}|y_{<n},x)\propto\exp(z_{n}/T)$ where $z_{n}=m_{\theta}(s_{n}|y_{<n},x)$ is the corresponding logit.
We also define $p_{\theta}(y|x)=\prod_{n=1}^{L}\pi_{\theta}(s_{n}|y_{<n},x)$ the probability of sampling the output sequence $y$ from input $x$.

### 2.1 CFG distillation for quality

CFG.
Given a model $m_{\theta}$ and a partial sequence $y_{<n}$, CFG is an inference-time strategy that samples the next token by combining conditional logits $m_{\theta}(s_{n}|y_{<n},x)$ (where $x$ is the user’s prompt) and unconditional/negative logits $m_{\theta}(s_{n}|y_{<n},x^{-})$ (where $x^{-}$ is an empty or negative prompt):

$$ $z^{CFG_{\gamma}}_{n}=\gamma\cdot m_{\theta}(s_{n}|y_{<n},x)+(1-\gamma)\cdot m_ {\theta}(s_{n}|y_{<n},x^{-}).$ (1) $$

The next-token probability distribution of the CFG-augmented policy is then
$\pi^{CFG_{\gamma}}_{\theta}(s_{n}|y_{<n},x)\propto\exp(z_{n}^{CFG_{\gamma}}/T)$.
The guidance factor $\gamma$ controls the adherence to the prompts; higher values typically lead to higher quality (Ho and Salimans, ). We use $\gamma>1$, extrapolating rather than interpolating, as done in Kreuk et al. ().
For example, if $x$ is “Soulful jazz song.” and $x^{-}$ is “Bad audio quality.”, a larger $\gamma$ increases the degree to which the generated sequence resembles typical elements of a soulful jazz song while increasing audio quality.

CFG distillation.
One key limitation is that CFG doubles the inference cost, since it requires the computation of two sets of logits.
To eliminate this overhead, we propose an objective that directly distills the benefits of CFG into the model weights.
Knowledge distillation (Hinton et al., ) traditionally involves compressing a large teacher model into a smaller student model (Sanh et al., ; Agarwal et al., ), allowing for efficient deployment while approximating the teacher’s performance.
In contrast, our teacher and student share the same architecture: the teacher (with weights $\theta_{\text{init}}$ from initialisation) is simply augmented with CFG.
We then follow the standard distillation approach that encourages the student to match logits from the teacher.

On-policy distillation.
If the teacher provides the label, the remaining question is: how to sample the input data? i.e., the completions on which teacher’s and student’s logits should match.
Be given a set of prompts, offline distillation uses inputs sampled from the teacher (Lin et al., ); in contrast, we adopt the on-policy distillation strategy from Agarwal et al. (), where data is online and dynamically sampled from the student itself. This reduces the train-test mismatch, a.k.a the exposure bias (Bengio et al., ), and was used in recent state-of-the-art LLMs (Gemma Team et al., )(^1^11Moreover, as we also encourage the generations from the student to optimise a diversity reward, samples from the student will be readily available, making this on-policy distillation possible at no cost.).

CFG distillation objective.
Overall, starting from the base pretrained initialisation $\theta_{\text{init}}$, we distill the logits obtained by $\theta_{\text{init}}$ with CFG into the weights $\theta$ without CFG by maximising:

$$ $\mathbb{Q}({\theta})=-\mathbb{E}_{x\sim X}\left[\mathbb{E}_{y\sim p_{\theta}( \cdot|x)}\left[\sum_{n=1}^{L}KL\left(\pi_{\theta}\left(\cdot|y_{<n},x\right)|| \pi^{CFG_{\gamma}}_{\theta_{\text{init}}}\left(\cdot|y_{<n},x\right)\right) \right]\right].$ (2) $$

### 2.2 RL for diversity

Reduced diversity during alignment.
Another key drawback of CFG is that it reduces the diversity of generated outputs as $\gamma$ increases.
This occurs because stronger guidance encourages the model to adhere closely to the prompt, leading to outputs that are more similar to each other.
This trend also emerges in our CFG distillation procedure (from Section 2.1). We confirm in Figure 2 (right) that diversity across generations from the student steadily decreases as training progresses.
More generally, policy collapse is a recurring challenge for alignment strategies; for instance, Kirk et al. () demonstrate that RLHF significantly reduces diversity of outputs.
This limits exploration of novel and diverse contents, a crucial aspect for creative domains.
To address this, we introduce a strategy encouraging the model to generate diverse samples.
This requires two components, detailed below: (1) a diversity reward and (2) a diversity-enforcing algorithm.

Diversity reward: negative cosine similarity of embeddings.
We first introduce a simple method to quantify the diversity between two generations $y_{1}$ and $y_{2}$. We embed those generations with a model $E$ and compute their cosine similarity in the embedding space.
Then, the diversity is defined as $r_{D}(y_{1},y_{2})=1-\frac{E(y_{1})\cdot E(y_{2})}{\lVert E(y_{1})\rVert\lVert
E
(y_{2})\rVert}$.
The key component of this diversity reward is thus the embedding model $E$.
In our text-to-music application from Section 3, inspired by Futeral et al. (), we train a novel embedding model $E$ using a contrastive learning approach. This involves training $E$ to map audio segments from the same song to nearby points in the embedding space while pushing apart embeddings of segments from different songs; this is further described in Section C.1.

Diversity algorithm: RL for diversity.
We now leverage reinforcement learning (RL) to encourage the student policy to generate diverse samples for a given prompt.
Using the diversity reward defined above, the diversity objective to maximise is:

$$ $\mathbb{D}({\theta})=\mathbb{E}_{x\sim X}\left[\mathbb{E}_{y_{1},y_{2}\sim p_{ \theta}(\cdot|x)}\left[r_{D}(y_{1},y_{2})\right]\right].$ (3) $$

As shown in Appendix A, an unbiased estimator of $\nabla\mathbb{D}(\theta)$ is then:

$$ $r_{D}(y_{1},y_{2})\nabla\log p_{\theta}(y_{1}|x)\quad\text{with}\quad x\sim X, y_{1},y_{2}\sim p_{\theta}(\cdot|x).$ (4) $$

Due to the similarity of Equation 4 with policy gradient (Sutton et al., ), this diversity can be optimised with standard algorithms such as REINFORCE (Williams, ), updating the parameters $\theta$ by increasing the likelihood of generations that lead to high (diversity) rewards.
Overall, the only difference is that this gradient asks for multiple generations, matching the requirements of multi-sample RL algorithms recently used in RLHF (Ahmadian et al., ; Sessa et al., ).

### 2.3 Diversity-rewarded CFG distillation

Combining both CFG distillation from Section 2.1 and the diversity reward from Section 2.2, we maximise the following novel diversity-rewarded CFG distillation objective:

$$ $\mathbb{QD}({\theta})=\mathbb{Q}({\theta})+\beta\times\mathbb{D}({\theta}),$ (5) $$

with $\beta$ the diversity hyperparameter scaling the RL diversity term to the KL quality distillation term.
Low values of $\beta$ (e.g., 0 0) lead to policies focused on quality, while large values of $\beta$ (e.g., $15$) lead to policies prioritising diversity at a slight cost in terms of quality, as visible in Figure 1 (right).

### 2.4 Model merging for Pareto-optimal quality-diversity

Tuning the hyperparameter $\beta$ allows for some control over the quality-diversity trade-off before training, but exploring the full spectrum of possibilities would require maintaining a large set of models with different $\beta$ values.
To address this, we leverage model merging, which enables combining the strengths of different models by simply interpolating their weights (Ilharco et al., ; Dimitriadis et al., ; Ramé et al., ). This allows us to efficiently adjust the quality-diversity trade-off at deployment time, based on the specific needs of the user or application, with only two finetunings.
Specifically, we consider two models finetuned from a shared pretrained initialisation with different values of $\beta$, $\theta_{q}$ focused on high quality ($\beta=0$) and $\theta_{d}$ focused on high diversity (high $\beta$).
We then trade-off between their abilities simply by interpolating between their weights:

$$ $\theta_{LERP}=(1-\lambda)\cdot\theta_{q}+\lambda\cdot\theta_{d},$ (6) $$

with $\lambda\in[0,1]$. For instance, setting $\lambda$ close to 0 0 favors the high-quality model, generating outputs that closely adhere to the prompt, while $\lambda$ close to $1$ favors the high-diversity model, resulting in more exploratory outputs.
This lightweight approach uncovers a strong and steerable quality-diversity front of solutions with minimal computational cost, similar to adjusting $\gamma$ in CFG during inference.

## 3 Experiments

The experiments below are structured to address the following research questions.
(1) CFG distillation: can we distill the quality of CFG into the weights of the model, and at what cost in terms of diversity?
(2) Diversity RL: can we trade-off between quality and diversity by including a diversity reward during finetuning?
(3) Model merging: can we construct at inference time a strong and steerable quality-diversity front of solutions with weight interpolation?

### 3.1 Text-to-music generation: task, setup and metrics

Task.
We explore those questions on text-to-music generation, a creative task where quality and diversity are important factors. Indeed, a single music prompt could map to many different valid generations. For instance, a gentle bossa nova piece, a relaxing acoustic guitar melody or an ethereal electronic soundscape would all be valid responses to the prompt “A peaceful sunset on a beach”. This makes text-to-music generation a great testbed to study quality-diversity trade-offs.

Model and setup.
We follow the experimental setup from Cideron et al. (), using their LLM transformer-based architecture, where a single autoregressive stage models the semantic and coarse acoustic features.
We use their MusicRL-R checkpoint as the base model $\theta_{\text{init}}$, initialising all our experiments.
We use the prompt dataset described in Section 4.1 from Cideron et al. (), combining multiple sources: synthetic prompts generated from the LaMDA model (Thoppilan et al., ), prompts collected via user interactions (Cideron et al., ), and prompts derived from MusicCaps (Agostinelli et al., ).
For CFG, we set $\gamma=3$ and use the negative prompt “Bad audio quality.”, as it performed best on the base model (cf. Appendix B).
For the diversity reward, we train a music embedding model $E$ by contrastive self-supervised learning (cf. Section C.1).

Training details.
The partial generations for distillation are sampled from the student in an online fashion (Agarwal et al., ) with temperature $T=0.99$.
The RL algorithm is a variant of REINFORCE (Williams, ) with a baseline for variance reduction.
We use a batch size of $128$ and a learning rate of $0.00015$ for all our finetunings.

Metrics. We evaluate the quality of the generations with two metrics: (1) the MuLan score (Agostinelli et al., ), a text adherence metric based on the MuLan embeddings (Huang et al., ), and (2) the User Preference score (Cideron et al., ), learned on 300k pairwise preferences of music generations to approximate the average human opinion score. We then define the general quality score as $s_{\text{quality}}=\omega\cdot s_{\text{MuLan}}+s_{\text{User Preference}}$, using $\omega=5$ to enforce a similar range of variations for the two scores.
Critically, those metrics are not used during training.
In Section 3.5, we confirm the insights obtained from this quality metric through human evaluation.

### 3.2 CFG distillation for quality

We want to validate whether or not it is possible to match by distillation (at training time) the performances of CFG-augmentation (at inference time) without its inference cost overhead.

CFG distillation optimisation.
Figure 2 (left) shows the evolution of the KL divergence from Equation 2 between the policy and the CFG-augmented policy along training.
When the objective is only to minimise the KL (i.e., $\beta=0$), the KL decreases to its lowest value, implying that the CFG-free model learns to approximate the CFG logits.
$\beta$ is scheduled to linearly increase until step $1000$, which explains why the KL increases after this step for $\beta>0$.

CFG distillation improves quality.
Figure 2 (middle) shows that training significantly improves quality matching the performances of the base model augmented with CFG and $\gamma=3$.
This suggests that CFG’s quality can be effectively distilled into the model’s weights, a finding confirmed by human evaluations in Section 3.5.

CFG distillation reduces diversity.
Figure 2 (right) shows that the CFG-distilled model ($\beta=0$) suffers from a decrease in diversity compared to its initialisation, the CFG-free base model.
Notably, after only $1$k training steps, CFG-distillation reaches the same level of diversity (around $0.37$) than the CFG-augmented base model (grey horizontal line for $\gamma=3$).
As a side note, Figure 1 (right), Figure 4 (right) and Figure 6 (in Appendix B) show that higher values of $\gamma$ for CFG further reduce diversity. In the next subsection, we assess the efficiency of our diversity reward to promote diversity.

Figure: Figure 2: Left. Evolution of the KL divergence between the CFG-distilled student and the CFG-augmented teacher along training. GKD distillation alone ($\beta=0$) decreases the KL between the two policies. Middle. Evolution of the quality along training, showing improved quality for all selected values of $\beta$. Right. Evolution of the diversity across generations along training, showing that CFG distillation alone reduces diversity, but that using a diversity reward ($\beta\neq 0$) can actually increase it. The “CFG” line shows the quality/diversity performance of the CFG-augmented base model serving as a teacher. The “upper-bound” line indicates the mean diversity of two generations (from the base model) for two different prompts.
Refer to caption: x3.png

### 3.3 RL for diversity

Rewarding diversity.
We now analyse the results obtained when including the diversity reward from Equation 3, scaled by the hyperparameter $\beta\in\{0,5,10,15\}$ in the joint objective.
As visible in Figure 2 (right), this increases the diversity across generations along training.
For $\beta=10$ or $\beta=15$, diversity actually gets larger, and closer to the mean diversity between two generations for two different prompts, denoted as an empirical “upper-bound”.

Quality-diversity trade-off.
Figure 2 (left) shows that these diversity gains come at the cost of increased KL.
This is because higher values of $\beta$ put more emphasis on the diversity term from Equation 3, hindering the minimisation of the KL distillation term from Equation 2.
Then policies finetuned with larger values of $\beta$ have lower quality in Figure 2 (middle).
This is summarised in Figure 1 (right), where quality-diversity trade-offs are plotted along training for different values of $\beta$.

### 3.4 Model merging for Pareto-optimal quality-diversity

Figure: Figure 3: Quality-diversity trade-off for multiple strategies. The first four lines linear interpolate (LERP) between the quality-focused model ($\beta=0$) and more diverse models (those trained with $\beta>0$, or the base model), sliding $\lambda$ between 0 0 and $1$ with a step of $0.05$. We also report the performance from the uniform ($\lambda=\frac{1}{4}$) averaging of the four models finetuned with different $\beta$, denoted as “LERP($0,5,10,15$) uniform”. We include inference-time baseline strategies — CFG (when varying $\gamma$) and temperature sampling (when varying the temperature $T$) — applied either on the base model or on the CFG-distilled model.
Refer to caption: x6.png

Quality-diversity trade-off.
In Figure 3 we interpolate between the model with highest quality (i.e., CFG-distilled with $\beta=0$) and models with more diversity: by sliding the coefficient $\lambda$ from 0 0 (only the quality model) to $1$ (only the diversity model), we construct fronts of solutions that surpass the performance of individual models finetuned for intermediate $\beta$ values.
In particular, interpolating towards the solution with maximum diversity ($\beta=15$) yields a strong and steerable Pareto front describing the full range of possible trade-offs.
This indicates that model merging achieves a higher quality for a given level of diversity, and vice versa, consistent with previous studies that show how merged models perform better than their non-merged counterparts (Wortsman et al., ; Ramé et al., ).
This is achieved with minimal overhead: only two RL finetuning runs are required, and merging the weights involves negligible computational cost.
If only one finetuning is possible, then linearly interpolating towards the base initialisation can also help, by recovering features from pretraining (Wortsman et al., ); this “LERP(0 0, base)” outperforms the CFG-augmented base model.
If more compute is available for training and we can do four finetunings, then merging them uniformly performs even better, as visible in Figure 3 where the grey cross for “LERP($0,5,10,15$) uniform” is above other fronts. This validates that we can merge an arbitrary number of models.

Figure: Figure 4: Left. Linear interpolation between the weights of a model focused on quality ($\beta=0$) and a model focused on diversity ($\beta=15$), sliding the coefficient $\lambda$ between 0 0 and $1$. The dashed diagonal represents the expected values if abilities were traded-off linearly between those two models. While the diversity stays close to the diagonal, the quality is above it, explaining the benefits of model merging. Right. For comparison, we also include the results for CFG when sliding $\gamma$ between $1$ and $7$, performing worse than merged models.
Refer to caption: x7.png

Baselines.
Figure 3 also displays the results for two inference-time baselines: CFG and temperature sampling. Applied to the base model, these strategies are Pareto-dominated by CFG-distilled models with diversity rewards. When applied to the already CFG-distilled model, they do not significantly improve quality nor diversity.

Quality-diversity as a function of $\lambda$.
Figure 4 (left) clarifies the effect of weight interpolation on quality and diversity as we slide $\lambda$ between 0 0 and $1$.
Diversity stays close to the diagonal (representing the expected diversity) while the quality is consistently above the diagonal.
This highlights the ability to significantly increase diversity with only minor quality compromises via model merging.

### 3.5 Human evaluation

Protocol.
To conclude our experiments, we validate via human evaluation the improved quality-diversity trade-offs. We evaluate five models: the base model (we expect low quality but high diversity), the base model with CFG $\gamma=3$ (high quality but low diversity), the CFG-distilled model for $\beta=0$ (high quality but low diversity) and $\beta=15$ (medium quality and high diversity), and finally LERP(0, 15) merging uniformly with $\lambda=0.5$ the two previous models (high quality and medium diversity).
For quality (i.e., acoustic quality, text adherence, and musicality), we use the same evaluation protocol as in Cideron et al. (): the raters see two generations from different models coming from the same text prompt and rate them on a scale from $1$ to $5$. We use $100$ different prompts and each one is seen by $3$ different raters.
For diversity, we rely on a similar protocol except that the raters see two pairs of music clips generated from the same text prompts and are rate how diverse the pairs of generations are. We use $50$ different prompts and each one is rated by $3$ raters.

Figure: Figure 5: Left. Side-by-side human evaluation for quality. Right. Side-by-side human evaluation for diversity. The score corresponds to the win rate of model A over model B, computed as $\frac{W+T/2}{W+L+T}$ with $W$ the number of wins of A over B, $T$ the number of ties, $L$ the number of losses of A against B. This confirms that our approach improves the quality-diversity trade-off. For instance, the merged model LERP(0, 15) generates music with higher diversity than the CFG-augmented base model ($\gamma=3$) in 57% of the comparisons, while being rated as more qualitative half of the time (51%).
Refer to caption: x11.png

Human evaluation for quality.
Figure 5 (left) presents the side-by-side win rate for quality. The CFG-distilled model (i.e., $\beta=0$) performs on par with the CFG-augmented base model (i.e., CFG) with a win rate of $0.50$; they both outperform the base model without CFG (first line), with $0.66$ and $0.68$ win rate respectively. LERP(0, 15) achieves win rate above $0.50$ against all models.

Human evaluation for diversity.
Figure 5 (right) presents the side-by-side win rate for diversity.
The CFG-augmented base model (i.e., CFG) and the CFG-distilled model (i.e., $\beta=0$) are also evaluated similarly in terms of diversity ($0.52$ win rate for the former); yet, they are consistently seen less diverse than the three other models.
Notably, the CFG-distilled model with diversity reward (i.e., $\beta=15$) has win rate of $0.73$ and $0.79$ against them.
As a side note, this $\beta=15$ model performs on par with the base model ($0.50$ win rate), though the quantitative metrics from Figure 2 (right) suggested that it was actually more diverse; in Section 5, we relate this discrepancy to the standard reward hacking (Amodei et al., ; Gao et al., ) phenomenon of RL.

#### Human evaluation for quality-diversity trade-off.

Human evaluations confirm the effectiveness of our approach in improving the quality-diversity trade-off.
The diversity-focused model ($\beta=15$) exhibits higher quality than the base model while maintaining comparable diversity.
Importantly, the merged model LERP(0, 15) exhibits higher diversity than the CFG-augmented base model while maintaining comparable quality,
and without incurring the additional inference cost of CFG.

Qualitative analysis.
To facilitate qualitative analysis, we host music generated from all evaluated models on the website
[google-research.github.io/seanet/musiclm/diverse_music](https://google-research.github.io/seanet/musiclm/diverse_music/).
An examination of generic prompts like “Rock song.” confirms that CFG improves quality (i.e., less acoustic artifacts, better musicality) but reduces diversity, as generated guitars tend to be very similar across different generations.
In contrast to the CFG-augmented model, the $\beta=15$ model generates a wider variety of rhythms while still demonstrating a clear improvement in quality over the base model.
On prompts like “Opera singer” or “Male choir harmony”, the quality-focused model $\beta=0$ generates conventional outputs, while the diverse model $\beta=15$ produces more unconventional and creative results, sometimes leading to unexpected elements like unusual instrumentation (e.g., drums). By averaging the weights of these two models (LERP), we can effectively balance these qualities, generating high-quality music that is both faithful to the prompt and more creative.

## 4 Related work

Classifier-free guidance (CFG). Initially introduced for diffusion models (Ho and Salimans, ) and later adapted for autoregressive LLMs (Gafni et al., ; Sanchez et al., ; Wings, ), CFG has found widespread application in various generative domains, including image (Yu et al., ; Saharia et al., ; Rombach et al., ; Nichol et al., ; Ramesh et al., ), video (Blattmann et al., ; Ho et al., ), and audio (Kreuk et al., ; Copet et al., ) generation. However, the detrimental impact of CFG on diversity is well-documented (Dhariwal and Nichol, ; Kreuk et al., ; Nichol et al., ), limiting its application when exploration is key.

Distillation.
Knowledge distillation (Hinton et al., ) is emerging as a powerful technique to train state-of-the-art models (Gemma Team et al., ).
By transferring knowledge from a teacher, the student can perform better than with standard training on the same data (Sanh et al., ; Lin et al., ; Gu et al., ).
Closer to our work, Meng et al. () propose a two-stage offline procedure to distill a CFG-augmented diffusion model.
In contrast, we employ a single-stage on-policy distillation procedure (Agarwal et al., ) to distill a CFG-augmented LLM, while introducing a novel diversity-promoting RL algorithm and model merging for improved quality-diversity trade-off.

Quality-diversity in LLMs.
Zhang et al. () compare the quality-diversity trade-offs of various inference-time strategies for LLMs, including temperature sampling, top-k sampling (Fan et al., ), and nucleus sampling (Holtzman et al., ). These methods perform similarly except when quality is prioritized over diversity, where nucleus sampling performs best.
Regarding finetuning strategies, Kirk et al. (); Mohammadi (); Chaudhari et al. (); Li et al. () have shown that RLHF can negatively impact the diversity of generated text, often measured with metrics like BLEU.
To the best of our knowledge, we are the first to introduce an RL algorithm to optimize diversity, which could potentially also solve those reductions in diversity in RLHF.
In contrast, existing quality-diversity algorithms (Lehman and Stanley, ; Mouret and Clune, ; Cully et al., ; Cideron et al., ; Ding et al., ) aim at finding a population of agents with both high-quality and diverse behaviors.
Most similarly, Li et al. (); Zhang et al. () also tried to increase the diversity across generations produced by a single agent, but measured by the number of distinct $n$-grams, and optimised with objectives based on mutual information (Shannon, ).

Model merging for Pareto-optimality.
Model merging via weight averaging (Utans, ) has two main applications in deep learning.
First, it increases robustness and generalisation by reducing variance (Wortsman et al., ; Ramé et al., ) and memorisation (Lin et al., ; Ramé et al., ).
Second, merging models combines their strengths (Ilharco et al., ; Dimitriadis et al., ; Wang et al., ), a property recently employed in Ramé et al. () for multi-objective RL, where policies finetuned with different rewards are interpolated. Similarly, we interpolate between policies where only one of them is rewarded for diversity.
Despite recent theoretical efforts to explain the empirical success of model merging (Ortiz-Jimenez et al., ; Ferbach et al., ; Ramé et al., ), a complete understanding remains elusive.

Music generation. Diffusion models (Huang et al., ; Schneider et al., ; Liu et al., ) and Transformers (Agostinelli et al., ; Copet et al., ) are now the state-of-the-art architectures for music generation.
In this work, we leverage the Transformer-based approach from Agostinelli et al. (), casting music generation as a categorical prediction task in the discrete token space provided by a neural audio codec (Zeghidour et al., ; Défossez et al., ).
RL-finetuning for music has previously been explored in Jaques et al. (); Guimaraes et al. (); Kotecha (); Latif et al. (); Cideron et al. ().
In contrast, we are the first to distill CFG, to enforce diversity through RL, to apply model merging or to improve the quality-diversity trade-off for music.

## 5 Discussions and limitations

Amplification and distillation.
In this work, we leverage a teacher-student framework where the teacher is an augmented version of the student model itself.
This allows to distill the quality improvements obtained from a potent but expensive inference strategy, eliminating the overhead at deployment.
This echoes the principles of iterated distillation and amplification (IDA) (Cotra, ), where the model iteratively learns on data generated by an augmented version of itself.
Examples of this can be seen in AlphaGo (Silver et al., ), achieving superhuman performance by distilling the knowledge of MCTS (Kocsis and Szepesvári, ), or in recent LLM works (Wang et al., ; Yu et al., ), distilling “System 2” into “System 1” by imitating offline predictions obtained from Chain-of-Thought (Wei et al., ).
Given the effectiveness of scaling inference-time strategies (Snell et al., ), we anticipate wider adoption of such amplification-then-distillation techniques.

Extension to other models or modalities. The three main components of our approach — distillation of an inference strategy, RL with a diversity reward, and model merging — could be readily adapted to other generative architectures.
For instance, model merging is already a popular strategy for diffusion models (Purplekeyboard, ; Biggs et al., ).
Additionally, while our focus is on text-to-music, similar strategies could be applied to other setups involving text, image or video generation.
In those domains, promoting diversity can foster fairer representations and pluralistic opinions (Srinivasan et al., ; Lake et al., ), but also facilitate exploring diverse reasoning paths (Liang et al., ; Yu et al., ) and enhance multi-sample decoding strategies (Bertsch et al., ; Li et al., ).
Diversity measures.
We used negative cosine similarity between embeddings as our diversity measure.
Critically, we demonstrate in Section C.2 its superior correlation with human perception of diversity compared to token-level entropy: we suspect this is because diversity should be assessed at the sentence level for creative tasks.
Yet, qualitative inspection suggests that this diversity measure can still be hacked (Amodei et al., ; Gao et al., ).
For example, models finetuned to maximise diversity sometimes generate excessive background drums, likely due to a bias of the underlying embedding model $E$.
This could be changed by making $E$ more invariant to background drums; alternatively, inspired by Ding et al. (), we could learn directly a human feedback (or AI-feedback) diversity embedding, by asking humans (or LLMs) to rate the similarity of generations.
More broadly, relying on a single diversity metric may be insufficient, as diversity can manifest in various ways (Levy et al., ); in music, diversity could focus on variations in voice, key, tempo, or other elements; in NLP, a summarization task may prioritise syntactic diversity over semantic diversity.
By incorporating multiple diversity rewards, each targeting a specific variation, and then leveraging model merging, we could achieve fine-grained control over diversity at deployment time.

## 6 Conclusion

In this work, we introduced diversity-rewarded CFG distillation, a novel finetuning approach to enhance the quality-diversity trade-off in generative models.
First, we online distilled CFG, eliminating its computational overhead at deployment.
Second, to enhance diversity across generations, we incorporated an RL procedure that optimises a diversity reward based on similarity embeddings.
Third, we leveraged model merging to enable dynamic control over the quality-diversity trade-off at deployment time.
Through extensive experiments on text-to-music generation, we demonstrated the validity of our strategy, with our finetuned-then-merged model performing best according to human evaluations.
We believe that our work is a promising direction for tasks where alignment and creativity are key, and that it can be easily extended to setups beyond text-to-music generation.

## Acknowledgements

We thank Daniele Calandriello and Nino Vieillard for insightful comments on drafts of the paper. We also thank the people involved in designing and building the distillation and RL training infrastructures used in our experiments: Léonard Hussenot, Robert Dadashi, Pier Giuseppe Sessa, Matt Hoffman, Abe Friesen, Andrea Michi, Sabela Ramos, Piotr Stanczyk, Danila Sinopalnikov, Amélie Héliou, Alexis Jacq, and Nikola Momchev.
Finally, we thank the contributors of MusicLM: Timo Denk, Mauricio Zuluaga, Marco Tagliasacchi, Matt Sharifi, Michael Dooley, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian
McWilliams, Victor Ungureanu, Ondrej Skopek, and Christian Frank.

## Appendix A Policy gradient for the diversity reward

In this appendix we consider the general case of optimising diversity across $N$ generations, where:

$$ $r_{D}(y_{1},\dots,y_{N})=\frac{\sum_{i,j\in\{1,\dots,N\},i\neq j}r_{D}(y_{i},y _{j})}{\sum_{i,j\in\{1,\dots,N\},i\neq j}1}.$ (7) $$

The key originality is that this reward considers multiple trajectories simultaneously, while the traditional policy gradient from Sutton () considers a single trajectory. We make the theoretical connection below.

Let’s consider a symmetric multi-trajectories reward, where the expected reward is defined as follows:

$$ $J({\theta})=\frac{1}{N}\mathbb{E}_{x\sim X}\left[\mathbb{E}_{y_{1},\makebox[7. 00002pt][c]{.\hfil.\hfil.},y_{N}\sim p_{\theta}(\cdot|x)}\left[r_{D}(y_{1}, \makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N})\right]\right].$ (8) $$

If we expand the inner expectation:

$$ $\mathbb{E}_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.},y_{N}\sim p_{\theta}( \cdot|x)}\left[r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N})\right ]=\sum_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.},y_{N}}r_{D}(y_{1},\makebox [10.00002pt][c]{.\hfil.\hfil.},y_{N})\prod_{i=1}^{N}p_{\theta}(y_{i}|x).$ (9) $$

Then we take the gradient:

$$ $\displaystyle\nabla\mathbb{E}_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.},y_{ N}\sim p_{\theta}(\cdot|x)}$ $\displaystyle\left[r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N}) \right]=\nabla_{\theta}\left[\sum_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.} ,y_{N}}r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N})\prod_{i=1}^{N }p_{\theta}(y_{i}|x)\right]$ (10) $\displaystyle=\sum_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.},y_{N}}\sum_{j= 1}^{N}\nabla p_{\theta}(y_{j}|x)r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil. \hfil.},y_{N})\prod_{{i=1,i\neq j}}^{N}p_{\theta}(y_{i}|x)$ (11) $\displaystyle=\sum_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.},y_{N}}\prod_{i =1}^{N}p_{\theta}(y_{i}|x)\left[\sum_{j=1}^{N}\nabla\log p_{\theta}(y_{j}|x) \right]r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N})$ (12) $\displaystyle=\sum_{j=1}^{N}\sum_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.}, y_{N}}\nabla\log p_{\theta}(y_{j}|x)r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil .\hfil.},y_{N})\prod_{i=1}^{N}p_{\theta}(y_{i}|x)$ (13) $\displaystyle=N\sum_{y_{1},\makebox[7.00002pt][c]{.\hfil.\hfil.},y_{N}}\nabla \log p_{\theta}(y_{1}|x)r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},y_{ N})\prod_{i=1}^{N}p_{\theta}(y_{i}|x).$ (14) $$

For Equation 12, we used $\nabla logx=\nabla x/x$. For Equation 14, we used the symmetry of $r_{D}$ and change of variable.
Hence, a non biased estimator of the policy gradient for $\nabla J(\theta)$ is:

$$ $r_{D}(y_{1},\makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N})\nabla\log p_{\theta} (y_{1}|x)\text{\leavevmode\nobreak\ where\leavevmode\nobreak\ }x\sim X,y_{1}, \makebox[10.00002pt][c]{.\hfil.\hfil.},y_{N}\sim p_{\theta}(\cdot|x).$ (15) $$

Due to the similarity of Equation 15 with the estimator of policy gradient for single-trajectory rewards, the diversity reward can be optimised with the usual policy gradient algorithms.
As an example, the sampled gradient with a batch size $B$ of the vanilla REINFORCE algorithm (Williams, ) would become
$\frac{1}{BN}\sum_{i=1}^{B}\sum_{j=1}^{N}r_{D}\left(y_{i,1},\makebox[10.00002pt
][c]{.\hfil.\hfil.},y_{i,N}\right)\nabla\log p_{\theta}\left(y_{i,j}|x_{i}\right)$ with $x_{i}\sim X,y_{i,j}\sim p_{\theta}(\cdot|x_{i})$.

## Appendix B CFG for text-to-music generation

Figure 6 shows the impact of CFG when applied at inference time on the MusicRL-R base model.
Specifically, we plot the quality-diversity front of solutions when sliding the adherence hyperparameter $\gamma$ for unconditional and negative prompting.
We observe that CFG significantly decreases diversity going from $0.5$ (for $\gamma=1$) to $0.3$ (for $\gamma=7$).
Critically, we also see that the front of solutions for negative prompting with “Bad audio quality.” is above the one revealed by unconditional prompting.
This specific negative prompt, inspired by those used in image generation (Mostaque, ), was selected based on preliminary experiments where it outperformed other negative prompt candidates.
Finally, for quality the best $\gamma$ is $3$, which we use throughout this work.

Figure: Figure 6: Effect of CFG on quality and diversity as a function of $\gamma$. The quality is greatly improved at the expense of diversity. Negative prompting outperforms unconditional prompting.
Refer to caption: x13.png

## Appendix C Diversity in music

### C.1 Embedding model for diversity

Diversity in music.
The objective of our diversity reward is to estimate the dissimilarity between two music segments. Diversity, especially in music, is a multifaceted concept encompassing various aspects such as instrumentals, melodies, tempo. In this work we conceptualise diversity through the following question: Are these two audio excerpts derived from the same audio recording or not?

Embedding.
Drawing inspiration from Futeral et al. (), we train a self supervised contrastive model with positive pairs coming from non-overlapping fixed-length chunks of the same audio segment.
The model induces an embedding space where segments originating from the same longer audio are mapped to nearby points, while segments from different recordings are mapped further apart.
This is useful because cosine similarity in the embedding space can quantify the similarity between two audio segments.
Overall, 1-similarity is used as the diversity measure.

Details.
Following Futeral et al. (), our embedding model takes as input a 4-seconds audio clip sampled at 16kHz. We generate the mel-spectrogram using a window size of 512 samples, a hop length of 256 samples, and 156 frequency bins. This results in 376 time frames, each with 156 dimensions.
The model architecture is founded on a compact Vision Transformer (ViT) (Dosovitskiy et al., ), comprising 12 layers. Each layer incorporates 6 attention heads, an embedding dimension of 512, and a feed-forward layer with a dimension of 1024 (totaling 25 million parameters). The final embeddings are averaged over time and then projected into a 192-dimensional space.
For training, we employ the semi-hard triplet loss (Schroff et al., ) with a total batch size of 3840, for 200,000 steps, using 16 kHz audio excerpts sourced from the same training dataset as Agostinelli et al. (). Additionally, we augment the training data with noise, tempo and pitch shifts.

### C.2 Comparison of our diversity metric with human evaluation and entropy

We compare the correlation between our diversity metric, the Elo score computed from human diversity evaluation in Section 3.5, and the entropy per generated token (which is a commonly used proxy for diversity (Zhang et al., )).
Figure 7 highlights that our diversity measure has higher correlation with the human raters’ notion of diversity than entropy.
As an example, $\beta=15$ scores high both in terms of Elo score and our diversity metric, while the entropy per token is as low as CFG (which is a low diversity approach).
Yet, our measure of diversity remains imperfect and thus can be hacked, as further discussed in Section 5.

Figure: Figure 7: Comparison between different measures of diversity; the one provided by our embedding model (left), the Elo rankings provided by human evaluation of diversity (middle) and finally the entropy per token (right). CFG at inference time (yellow) reduces all those measures of diversity. Models trained with our RL diversity reward ($\beta=15$ in green) have increased diversity according to our embedding model and human evaluation, but not in terms of entropy.
Refer to caption: x14.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: About music embedding
Why not use the open-source music embedding models to facilitate the reproducibility of this work? 

Are there any results of embedding evaluation like acc of linear probing that can be public for the music embedding model?
soundness: 2
strengths: Originality: The paper presents an innovative approach to overcoming CFG’s limitations, particularly by combining distillation with a diversity-promoting reinforcement learning reward. It proposes a metric of diversity based on pair-level dissimilarity. 
Quality: The methodology is robust, with well-designed experiments that rigorously evaluate quality-diversity trade-offs. The use of human evaluations adds significant credibility, and the proposed model-merging strategy for adjustable quality-diversity control demonstrates strong, reliable results.
Clarity: The paper clearly articulates its objectives, methods, and findings, making complex topics accessible. Detailed diagrams and well-structured explanations approach and results easy to understand, enhancing the overall readability and impact.
Significance : By improving both quality and diversity without increasing inference cost, this work addresses an important limitation in generative models. While primarily applied to music generation, the approach has potential implications across other generative domains, offering a valuable contribution to fields where creativity and efficiency are essential.
summary: This paper introduces a fine-tuning method that enhances both quality and diversity in generative models, targeting CFG’s limitations in computational cost and diversity reduction. The authors propose a two-part training objective: distilling CFG’s quality-enhancing effects into model weights and using reinforcement learning to encourage diversity. They also introduce model merging, which allows flexibility in balancing quality and diversity at deployment. Experiments on MusicLM show improvements over CFG, with human evaluations confirming superior quality-diversity trade-offs.
weaknesses: About the diversity training
According to equation 3, D(\theta) dependents on the expected value of dissimilarity of y given x. How many ‘y’s  are sampled given an x ? Besides, this paper does not further investigate the impact of the sample number of an x on the diversity training . 


The definition of diversity is based on pair-level dissimilarity. It may be better to use some metrics like gini or Kurtosis to measure the global diversity of the generated songs.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Rewarding to Weakness 2, would you mind including FAD or other metrics to state the quality? 
2. "Diversity" is conceptualized as "Are these two audio excerpts derived from the same audio recording or not?". While this diversity is essential to study and learn, since we are dealing with music, is that possible that we can have better conceptualization of "Diversity"? For example, "Given one reference audio recording, how diverge are these two audio excerpts in terms of genre comparing to the reference audio?" In short, I think the word "Diversity" can be more music related.
3. Same logic applies to "Quality". Audio quality can be applied to speech as well. Since this work only focuses on music, can "Quality" be more music focused? For example, the structure and key coherence in the short excerpt. 
4. Again, RL with a diversity reward objective seems like a general objective used or inspired from other fields, like population-based RL. While the application to music is novel, I don't think this is novel enough to be highlighted as novelty. Would you mind defending your argument of novelty? Why this objective special to music (this objective can be applied to speech as well)?
soundness: 3
strengths: 1. This work provides a new strategy for managing the quality-diversity balance in generative models and has potential applications in other areas of creative content generation.
2. To achieve both diversity and quality, instead of doing direct multi-objective training which can downgrade the performance, the authors first distill model in terms of audio quality with classifier free guidance, which increase the quality while maintaining the capability of text understanding to music.
3. The analysis of quality-diversity tradeoff provides further insights to music generation models, where quality and diversity are actually two factors that need to be balanced.
4. Great demos and great results! Did not listen to all of them, in particular, I like LERP 50/50 of “Melodic Danceable Brazilian Music with Percussions” the most.
summary: The paper proposes a method called Diversity-Rewarded Classifier-Free Guidance (CFG) Distillation for improving generative models, particularly in creative tasks like music generation. The authors introduces two training objectives:
1. Distillation: The method distills CFG’s quality improvements directly into the model weights, which eliminates the extra computational overhead during inference.
2.Reinforcement Learning (RL) for Diversity: A reinforcement learning objective rewards the model for producing diverse outputs, balancing the quality-diversity trade-off.
The approach focuses on a text-to-music generation task, where it successfully improves quality and diversity, validated by human evaluations.
weaknesses: 1. For music, the terms “quality” and “diversity” are not obvious in the paper. “Quality” can not only refer to audio quality, but can also refer to other aspects of music, like style consistency, the key consistency, rhythmic patterns, etc. I assume “quality” in this paper means the audio quality only. "Diversity" is specified in Appendix C.1, which I think it "Quality" and "Diversity" should be mentioned clearly in the introduction or methodology section.
2. In terms of evaluation of music, using MuLan score is great in terms of text coherence of generated music, while assuming “quality” means audio quality, it would be nice to include Fréchet Audio Distance (FAD) (Kilgour et al., 2019).
3. RL with a diversity reward objective maybe is newly applied to the realm of music, but this is not new in the RL world. I believe at least the work “Effective Diversity in Population Based Reinforcement Learning” (Parker-Holder et al., 2020) is worth mentioning and I did not see this work is cited in the current paper.  I would like to see some editing in the paper to change the wordings. If I miss this citation or relevant citations, please point out and correct me.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Line (254-255): In discussing batch size, is this for the number of samples or the number of pairs samples generated? In general, it is not clear when discussing the RL objective how these pairs are chosen in the context of training, and whether the RL objective is only calculated on single pairs of samples or multiple pairwise comparisons.
- In Figure 3, is there a reason there is only a single LERP data point for the 4-way merge? It would be useful to see if further interpolation between the 4 models is able to achieve a strictly dominant Pareto front above the LERP(0,15) model, as the claim that “This validates that we can merge an arbitrary number of models.” (Line 359) seems somewhat under-validated in this context.
- Line 962, what is 1-similarity?
- While certainly not important, I wonder if the authors have insight as to how techniques from LLMs in the contrastive decoding literature (especially works like DoLa which are self-contrastive) are related to CFG for autoregressive LM TTM models. In fact, the update equations from CFG and contrastive decoding/DoLa are practically identical, as they both involve the log difference in probabilities between the base model and some “worse” reference model.
soundness: 3
strengths: - Proposed CFG distillation method seems strong and is reasonably novel in the context of autoregressive LM models
- As a whole, proposed method is exceedingly clear, and while simple, shows the strengths of each facet of their 3-pronged method articulately.
- The significant time dedicated to discussing results and ablations is particular welcome, and the authors do a great job of exploring the design space of their method.
- The insight from Fig. 4 is in particular quite fascinating, do you have any idea why this merging behavior actually improves quality beyond the interpolating models?
summary: In this work, the authors introduce a novel finetuning recipe for increasing the runtime efficiency of the model and improving diversity at the same time, through Diversity-rewarded CFG distillation and model merging. The authors show that by distilling the CFG behavior into the model with an RL-based diversity regularizer and then merging high diversity with high-quality model finetunes, they can achieve strong diversity and quality tradeoffs and improved results over the base model.
weaknesses: Overall, while I think this is a reasonably strong work, there are a number of concerns I have addressed below that shift my overall recommendation to reject. In fixing some or all of these issues (in particular improving the discussion of CFG distillation novelty, using publicly available eval metrics such as FAD/CLAP/Recall/Coverage, including human eval error bars / statistical tests, and focusing more on the diversity model and its calibration in the main text) I would raise my score accordingly.

# Overclaiming of CFG Distillation Novelty:
In general, while the core contribution of the CFG distillation approach is interesting, I think it’s stated novelty is an over claim (such as the line 492-493 “In contrast, we are the first to distill CFG…for music,” which is false). While CFG Distillation may be novel in the case of autoregressive audio generation (MusicLM, MusicGen), CFG distillation is a common practice in diffusion models for the same reasons motivated in this paper (namely, reducing inference cost), normally coupled with other distillation techniques. While the authors rightly mention Meng et al., this work is rather old and the stated novelty of their “two-stage” approach vs. the authors’ “single-stage” approach (461-462) does not really hold, as most works after Meng et al. (and before the present work) use a single stage approach (and in general, the paper is presented as if the whole of CFG distillation is a fully novel contribution). Single-stage CFG distillation has been a standard practice in image domain distillation [1, 2], but more saliently has appeared in a number of audio (including music) generation [3,4] and music-specific generation [5] distillation works. As an actionable change, it would be useful to explicitly mention all of these connections in the paper (in both the introduction, the related works, and the section on CFG distillation), and make clear that the novelty of the proposed method comes from how they achieve CFG distillation (and the first to do so on autoregressive LM-based music models), rather than the *idea* of CFG distillation for music generation.


# Reduced Discussion on Diversity Model:
- It is odd that perhaps the most novel portion of the paper of the RL Diversity Objective (as both CFG distillation and model merging are standard practices) is relegated to most details of the core embedding model being in the appendix. To me, this is a core part of the paper (as defining diversity through some model underpins the entire method, and is very interesting), and it would be useful to include this in the main body of work (and I think there is a good deal of text in the results section that is redundant and could be condensed). 

# Issues with Evaluation Suite:
- While the authors make no mention of open sourcing model weights or code (which in my opinion is fine and shouldn’t be counted as a negative against them), the overall evaluation suite **also** uses fully closed-source models, severely limiting the reproducibility of the work and how further research could use it as a baseline. While I am sure that the MuLan embeddings and User Preference score are fine metrics in their own right, they are wholly inaccessible to the wider community. To accurately assess how the proposed method performs, not only to their own baselines but also external models, it is imperative to include metrics that are publicly usable in the literature, such as CLAP score to assess text-relevance, FAD/MMD (with CLAP backbone) to assess audio quality w.r.t. a reference set (like SongDescriber), and so on. I very much recommend the authors include some number of these (such as FAD and CLAP) in order to contextualize their performance in the larger space of TTM generation. In parallel, running existing open source models (such as AudioLDM2 or Stable Audio Open) on the present closed source eval suite would also work.
- The definition of “quality score” (259-260) is somewhat divorced from the literature on TTM and generative evaluation. This seems to combine MuLan score (theoretically a measure of “text relevance”) and user preference, but this explicitly entangles these two different axes of behavior that are generally kept separate (as how previous works do not combine CLAP and FAD for instance [3,4]). It should either be explicitly stated as to *why* this combination can be made, or more explicitly simply uncombine them and present all results for user preference and MuLan score separately, as currently it is impossible to tell *which* of these metrics the proposed method actually improves.
- I have strong worries on the overall statistical power of the human eval. As the human eval is a core part of the presented results, both in order to prove the efficacy of the proposed method and to back-verify that the diversity model correlates with human perception, it is unclear what sort of statistical significance actually occurs here, as there seem to be no hypothesis tests run nor error bars reported for any of the human eval metrics, exacerbated by the fact that there seems to be only 3 users in the listening study (which is far lower than even the small sample sizes common in generative media research). Specifically, it would be useful to see paired t-tests run between each method on the scores (averaged across human raters) to determine significance.
- There is some circularity to the evaluation of diversity in this work, despite it being a core focus. Though the authors rightly identify that the quality score is a good measure as it is not used in training, using the diversity model for eval and RL fine-tuning by nature biases results towards the proposed method (as it directly optimizes for this result). This ignores the fact that there are a number of established different metrics for assessing diversity in generative media, including Inception Score (IS), Recall, Coverage, and Vendi Score to name a few. Including some or any one of these scores would be useful both to bring it more inline with existing TTM literature as well as assess the output diversity of the model in an unbiased fashion and help verify whether the trained diversity model correlates with these values.
- Along with the above 2 points, the overall justification of the diversity model’s success hinges on Figure 7. (Middle)’s analysis that the fine-tuned model shows similar Elo ranking to the base model. I’m not sure if Elo score is the best measure for this, as it is a purely relative measure and details on its exact calculation are not provided.  It would be better to show the *absolute* scores for each model (with error bars) and see how well this correlates with the diversity model, and in general justify more in the paper that the diversity model is reasonably calibrated given its underpinning for the whole work.

[1] Luo, Simian et al. “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference.” ArXiv abs/2310.04378 (2023): n. pag.

[2] Yin, Tianwei et al. “Improved Distribution Matching Distillation for Fast Image Synthesis.” ArXiv abs/2405.14867 (2024): n. pag.

[3] Bai, Yatong et al. “ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation.” Interspeech 2024 (2023): n. pag.

[4] Saito, Koichi et al. “SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound Generation.” ArXiv abs/2405.18503 (2024): n. pag.

[5] Novack, Zachary et al. “DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation.” ArXiv abs/2405.20289 (2024): n. pag.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Please refer to weaknesses.
soundness: 2
strengths: The paper is extremely well written and the motivation is very clear. It aims to solve two very important challenges in text-to-music generation: inference overhead and bad quality-diversity tradeoff. The proposed method makes a concrete understanding towards these challenges.

The methods including CFG distillation, diversity maximization, and the use of model weight interpolation are straighforward and simple enough that these ideas could be potentially easily and widely adapted to the deep generative model community. 

The experimental results show concrete improvements on the quality-diversity tradeoff compared to the base model with or without CFG.
summary: The paper proposes a distillation based method to improve text to music models (specifically MusicLM and MusicRL). The student network learns the CFG-modified logits of the teacher model, and therefore running the teacher model with CFG is approximated with running the student network directly, which reduces inference compute to half. To address less diversity caused by CFG and by the CFG distillation, the paper proposes to maximize the diversity computed from inter-similarities during CFG distillation. The paper also finds directly interpolating two model weights computed from different $\beta$ leads to better quality-diversity tradeoff.
weaknesses: The major weakness of the CFG distillation is that it destroys the student model's ability to take different CFG scale as input, which is key in many downstream applications. I understand that this could be useful in applications where the inference strategy (i.e. CFG) is fixed by nature, but in many cases we need the flexibility to take different CFG scales at inference time. Therefore, I would expect the student model to take the CFG $\gamma$ as inputs as well, which could be done in a similar way how diffusion models are conditioned on timesteps.

The second weakness is that in Figure 5, the proposed model does not outperform CFG significantly. The quality is about the same, and the diversity is 57-vs-43. In addition, the diversity is much worse than the base model after model weight merging, indicating that the merged model still suffers from lack of diversity. 

The third weakness is that the paper does not evaluate the proposed distillation method on other baseline models and tasks (e.g. audio generation and speech generation) -- especially open-sourced models -- which limits the contribution of the paper. As a result, it is unclear how general the proposed method is when applied in other data and tasks. 

Minor weaknesses:

One minor weakness is that the paper does not analyze in depth the diversity drop of the vanilla CFG distillation. From Fig 2 one could infer it is most likely because the student model learns the CFG-modified logits well, but it is also possible that the KL loss in eq (2) amplifies it. It would be useful to study different divergence losses and different model capacities.

Another minor weakness is that Fig 3 is almost impossible to read. Please improve the display of this figure.

Meta-review: This paper introduces a fine-tuning method that enhances both quality and diversity in generative models, targeting classifier-free guidance’s limitations in computational cost and diversity reduction. 

All reviewers described originality, clear description, and promising results with demonstration as the strenghths with borderline acceptance scores. Although reviewers pointed out some issues such as metrics and unclear quality definition, they seem to be not major issues.

In particular, this work addresses a music generation task, which is relatively less prevalent research work.

Considering the contributions and topic diversity, AC recommends accepting this paper.

Decision: Accept (Poster)