SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks

## Abstract

Abstract Preference Optimization (PO) has proven an effective step for aligning language models to human-desired behaviors. Current variants, following the offline Direct Preference Optimization objective, have focused on a strict setting where all tokens are contributing signals of KL divergence and rewards to the loss function. However, human preference is not affected by each word in a sequence equally but is often dependent on specific words or phrases, e.g. existence of toxic terms leads to non-preferred responses.
Based on this observation, we argue that not all tokens should be weighted equally during PO and propose a flexible objective termed SparsePO, that aims to automatically learn to weight the KL divergence and reward corresponding to each token during PO training. We propose two different variants of weight-masks that can either be derived from the reference model itself or learned on the fly.
Notably, our method induces sparsity in the learned masks, allowing the model to learn how to best weight reward and KL divergence contributions at the token level, learning an optimal level of mask sparsity.
Extensive experiments on multiple domains, including sentiment control, dialogue, text summarization and text-to-code generation, illustrate that our approach assigns meaningful weights to tokens according to the target task, generates more responses with the desired preference and improves reasoning tasks by up to 2 percentage points compared to other token- and response-level PO methods.

## 1 Introduction

The rise of employing Large Language Models (LLMs) as conversational agents has increased the importance of aligning them with human preferences. Preference Optimization (PO), i.e. the training paradigm that aims to steer models to a desired behavior (typically related to human perception), is considered the last and most important step in the pipeline of LLM training for producing accurate, harmless and controllable models.
Reinforcement Learning from Human Feedback (RLHF; Christiano et al. ()) was the primary method for obtaining such a behavior. However, due to it’s inherent complexity it has been overpowered by Direct Preference Optimization (DPO) (Rafailov et al., ), a simpler, offline approach that produces a policy model that fits the preference data without the need for reinforcement learning.

DPO performs at the sequence level, optimizing rewards and measuring KL divergence for complete responses. However, various studies have shown that signals from specific tokens are primarily responsible for learning desired behaviors, both during pre-training (Lin et al., ) and preference optimization (Yang et al., ).
In particular, in domains where the preference is determined by a specific aspect (e.g. sentiment, toxicity) or when the decision relies on certain subsequences (Pal et al., ), it is necessary to consider more fine-grained updates. To further illustrate this point, Figure 1 shows that DPO is already learning implicitly to assign different token-level rewards, with higher values on a few tokens with positive/negative polarity (e.g. pretty, weak).
However, noting the various lone tokens with high rewards, DPO’s reward distribution seems inconsistent, and we posit that it would benefit from a more explicit signal.

Figure: Figure 1: Token-level rewards for chosen (top) and rejected (bottom) responses given an input prompt. After a GPT2-Large model is trained with DPO on the IMDB dataset to generate positive movies reviews, these rewards are calculated as the log ratio of token probabilities between policy (DPO) and reference model (original GPT2-Large). Denser values indicate higher probability score assigned to a token by the policy than the reference, implying importance towards that preference.
Refer to caption: extracted/5910009/dpo_imdb_chosen_rew.png

Aligned with prior work, we argue that not all tokens are important in preference optimization. We further propose that in order to have more diverse responses, and flexible optimization, we should allow only certain tokens to be close to the reference model so that the rest are able to grow beyond it--dismissing the need for measuring KL divergence on all tokens.
As such, in this work we propose sparse token-level preference optimization (SparsePO), a method that learns automatically during training inherently sparse masks over token-level rewards and KL divergences.
Approaches that have been developed based on this observation, either use external models to identify important tokens (Yoon et al., ) or need to first perform DPO training to select high-rewardable tokens (Yang et al., ).
Our method targets flexibility, with masks that can be either shared or independent between rewards and KL divergence. In addition, it is not reliant on external models and can be combined with any possible masking method. In this work, we present two masking strategies but any masking over tokens can be used instead.

Our contributions include (1) a flexible framework, termed SparsePO, for weighting token-level reward and KL contributions tailored to the offline preference optimization objective, (2) analyses over the induced masks’ sparsity and reward frontier and how they correlate with controlled KL divergence, (3) quantitative and qualitative gains when employing our proposed approach to different domains with explicit or implicit preference indicators.

## 2 Methodology

### 2.1 Preference Optimization

The purpose of aligning models with human preferences is to steer model behavior to produce human-acceptable responses.
To realize that, we assume training data in the form of static, paired preferences. A prompt $x$ is associated with two responses, chosen $y_{c}$ and rejected $y_{r}$, so that $y_{c}$ is preferred over $y_{r}$ ($y_{c}\succ y_{r}|x$), resulting in a dataset $D=\{x^{(i)},y_{c}^{(i)},y_{r}^{(i)}\}_{i=1}^{N}$. Such responses and their rankings are typically collected either by humans or automatically from other models (Xu et al., ).
In PO, we aim to train a model to generate responses closer to $y_{c}$ than $y_{r}$.

In the standard Reinforcement from Human Feedback (RLHF) pipeline (Ziegler et al., ) this is realized in a sequence of steps.
Firstly, we perform supervised fine-tuning on the task for which we would like to learn preferences, to shift the distribution of the language model in-domain with the PO data.
Then, a reward model is trained, responsible for assigning a higher score (reward) to chosen responses and lower scores to rejected ones.
Given a policy network $\pi$ (i.e., the model that we aim to optimize), responses are sampled and then scored by the reward model. The policy training aims to maximize the rewards associated with chosen responses and minimize those of rejected ones, subject to a KL constraint with a reference model $\pi_{\text{ref}}$. The constraint prevents the policy $\pi$ from deviating too much from the distribution that the reward model has learned, as well as avoids reward hacking.
The above process is translated into the following objective.

$$ ${J}_{\pi}=\max_{\pi}\mathbb{E}_{x\sim D,y\sim\pi(\cdot|x)}\left[r(x,y)\right]- \beta\;D_{\mathrm{KL}}\left[\pi(\cdot|x)\|\pi_{\text{ref}}(\cdot|x)\right],$ (1) $$

where $r(x,y)$ corresponds to the reward for response $y$ given input $x$, $D_{\mathrm{KL}}$ is the Kullback-Leibler Divergence between the policy $\pi(\cdot|x)$ and the reference model $\pi_{\text{ref}}(\cdot|x)$ over response sequences.
In practice, the policy and reference models are the same at the beginning of training while the latter remains frozen.

### 2.2 Sparse Preference Optimization

Motivated by the fact that not all tokens are required to infer a preference, and in order to control token-level contributions, we start by converting the previous objective (Equation 1) that operates on the sequence-level to token-level. Based on the work of Zeng et al. () (TDPO), this corresponds to maximizing the following equation:

$$ $J_{\pi}=\max_{\pi}\mathbb{E}_{x\sim D,y^{t}\sim\pi(\cdot|x,y^{<t})}\left[A_{ \pi_{\text{ref}}}(y^{t}|x,y^{<t})\right]-\beta D_{\mathrm{KL}}[\pi(\cdot|x,y^{ <t})||\pi_{\text{ref}}(\cdot|x,y^{<t})]$ (2) $$

with $A_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})\equiv Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t}
)-V_{\pi_{\text{ref}}}(x,y^{<t})$ being the advantage function for the reference model as the difference between the state-action $Q$ and the state-value function $V$, and $\beta$ being a tunable parameter controlling the deviation from the reference model. Note that here the KL divergence is over the next-token distribution (i.e., vocabulary).

We argue that in order to control the contribution of each token, we can add a weight in front of the token-level KL divergence term, so that not all tokens are forced to stay close to the reference model. We speculate that this will lead to more diverse generation of responses, since only a handful of important tokens that indicate preference will have to be in-distribution.

Thus, we introduce a mask function $m(y^{<t})\in[0,1]$, $m(y^{<t})>\epsilon$ that produces a scalar for each token $y^{t}$ in a sequence $y$ that measures the amount of token KL participation in the loss function.

$$ $J_{\pi}=\max_{\pi}\mathbb{E}_{x\sim D,y^{t}\sim\pi(\cdot|x,y^{<t})}\left[A_{ \pi_{\text{ref}}}(y^{t}|x,y^{<t})\right]-\beta\;m(y^{<t})\;D_{\mathrm{KL}}[\pi (\cdot|x,y^{<t})||\pi_{\text{ref}}(\cdot|x,y^{<t})]$ (3) $$

Deriving Equation 3, in a similar manner as TDPO, and assuming that the mask is dependent on the reference model alone and on previously seen tokens, $m(y^{<t})=f_{\pi_{\text{ref}}}(x,y^{<t})$, we end up with the below optimal policy (refer to Appendix A.1 for a detailed solution),

$$ $\pi^{*}(y^{t}|x,y^{<t})=\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}(y^{t}|x,y^{<t}) \exp\left(\frac{1}{\beta\;m(y^{<t})}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})\right),$ (4) $$

where $Z(x,y^{<t})$ is the partition function.

The Bradley-Terry model (Bradley & Terry, ) is a popular theoretical formula employed to model the human preference distribution. As it operates on the sequence-level, its equivalent to the token-level is the Regret Preference model as previously proven by Zeng et al. ().

$$ $P_{BT}(y_{c}>y_{r}|x)=\sigma\left(\sum_{t=1}^{T_{1}}\gamma^{t-1}A_{\pi}(y_{c}^ {t}|x,y_{c}^{<t})-\sum_{t=1}^{T_{2}}\gamma^{t-1}A_{\pi}(y_{r}^{t}|x,y_{r}^{<t} )\right).$ (5) $$

Solving Eq. 4 for $Q_{\text{ref}}$, considering $A\equiv Q-V$ and substituting to Eq. 5, we obtain the final objective, named SparsePO. Our primary difference is that $m$ is dependent on each token effectively weighting both components of the objective (refer to Appendix A.2 for the detailed solution).

$$ $\mathcal{L}_{\text{SparsePO}}=-\mathbb{E}_{x,y_{c},y_{r}\sim D}[\log\sigma \left(u(x,y_{c},y_{r})-\delta(x,y_{c},y_{r})\right)]$ (6) $$

$$ $\displaystyle u(x,y_{c},y_{r})$ $\displaystyle=\beta\sum_{t=1}^{T_{1}}{m_{u}(y_{c}^{t})}\log\frac{\pi^{*}(y^{t} _{c}|x,y^{<t}_{c})}{\pi_{\text{ref}}(y^{t}_{c}|x,y^{<t}_{c})}-\beta\sum_{t=1}^ {T_{2}}{m_{u}(y_{r}^{t})}\log\frac{\pi^{*}(y^{t}_{r}|x,y^{<t}_{r})}{\pi_{\text {ref}}(y^{t}_{r}|x,y^{<t}_{r})}$ (7) $$

$$ $\displaystyle\delta(x,y_{c},y_{r})=\beta D_{\text{MaskKL}}[x,y_{c};\pi^{*}\| \pi_{\text{ref}}]-\beta D_{\text{MaskKL}}[x,y_{r};\pi^{*}\|\pi_{\text{ref}}],$ (8) $$

where $D_{\text{MaskKL}}[x,y;\pi^{*}\|\pi_{\text{ref}}]=\sum_{t=1}^{T}{m_{d}(y^{t})}
\;D_{\mathrm{KL}}[\pi^{*}(\cdot|x,y^{<t})\|\pi_{\text{ref}}(\cdot|x,y^{<t})]$.
The objective effectively adds token-level masks $m_{u}$ on rewards (Equation 7) and $m_{d}$ on the KL (Equation 8) for each response respectively.
Naturally, these masks can either be shared or be independent. In the following sections we experiment with both $m_{u}=m_{d}$ and $m_{u}\neq m_{d}$.

### 2.3 Mask Computation

In the previous section we showed how we can control the contribution of rewards and KL divergence of each token through the introduction of weights in the loss function.
Next, we introduce two strategies to obtain these weights from the reference model,
one that is derived directly from its internal activations and another that is learned in parallel during preference optimization.

Model Activation-based Mask (MaPO)

Inspired by mechanistic interpretability approaches (Huben et al., ), we leverage the rich information captured per token in the activations of the reference model and aggregate them into token-level weighting masks, as follows.
Let $a^{t}_{g}\in R^{d^{\prime}}$ be the output of activation function $g(*)$ in network $\pi_{ref}$,
and $\bar{a}^{t}_{g}$ its average value across dimensions for time step $t$.
Note that $a^{t}_{g}$ is exposed to information from $y^{<t}$ due to the autoregressive nature of generation.
We obtain
${[\tilde{a}^{1}_{g},..,\tilde{a}^{T}_{g}]}$,
where ${\tilde{a}^{t}_{g}=(\bar{a}^{t}_{g}-\operatorname*{mean}(\bar{a}_{g}))/std(
\bar{a}_{g})}$ is the standardization
of $\bar{a}$ across sequence $y$.
Then, we define activation-based mask ${m(y^{<t})=\operatorname*{mean}\{\tilde{a}^{t}_{g}|\forall g\in\pi_{ref}\}}$, i.e. the average $\tilde{a}^{t}_{g}$ for all activations in the reference model.
In practice, we aggregate outputs from feed-forward layers, residual connections, and attention layers, across all layers in $\pi_{ref}$.
Finally, we set $m_{u}(y^{<t})=m_{d}(y^{<t})=m(y^{<t})$, i.e. a common mask for the rewards and KL terms given.

Learnable Sparse Mask (SparsePO)

In our second variant, mask $m(y^{<t})$ is computed using learnable parameters. Specifically, we learn one feed-forward network (FFN) with ReLU activation for each model layer, and aggregate representations from all layers with a linear layer.(^1^11We initially experimented with learning two FFNs per layer, one for the chosen and one for the rejected responses. However this led to overfitting, hence we learn a single vector per layer.)
A single layer mask is computed as follows:

$$ $m^{(l)}(y^{<t})=ReLU\left(\mathbf{H}^{(l)}(y^{t})\cdot\mathbf{w}^{(l)}+\mathbf {b}^{(l)}\right),$ $$

where $\mathbf{H}^{(l)}\in\mathbb{R}^{N\times d}$ corresponds to the reference model hidden representation for layer $l$ for $N$ tokens and $\mathbf{w}^{(l)}\in\mathbb{R}^{d},\mathbf{b}^{(l)}$ are the $l$-layer learned parameters.
Consequently, when learning multiple masks per layer, they are combined as

$$ $m(y^{<t})=ReLU\left(\text{Concat}\left(m^{(1)}(y^{<t}),...,m^{(L)}(y^{<t}) \right)\cdot\mathbf{w}_{o}\right),$ $$

with $\mathbf{w}_{o}\in\mathbb{R}^{L}$ the output merging vector.

The ReLU activation function produces a sparsity in the masks, the degree of which is dependent on the target preference data and the reference model.
The mask values (independent of strategy) are utilized solely during PO training and are ignored during model inference.

## 3 Experiments

In this section, the effectiveness of SparsePO is investigated in both proxy-preference and human-preference setups.
Proxy-preference setups are analyzed through sentiment control, summarization, and code generation,
whereas human-preference setup is analyzed through single-turn dialogue tasks.
We refer the reader to Appendix B for further details on experimental setup.

### 3.1 Model Comparison

We compare the performance of SparsePO against supervised fine-tuning over preferred responses (SFT, serving both as a baseline and the starting point of the PO variants) and performant PO strategies that model preference at the sequence and token levels.
At the sequence level, we compare against DPO (Rafailov et al., ), which aims to mitigate KL divergence;
SimPO (Meng et al., ), which aims to maximize the probability difference between chosen and rejected responses;
and DPOP (Pal et al., ), which adds a penalty term to the DPO loss to encourage high probability scores of the preferred completions.
At the token level, we compare against TDPO v1 and v2 (Zeng et al., ), which adds token-level KL divergence as a regularization term.
Unless stated otherwise, we investigate SparsePO setups learning a common mask for reward and KL terms ($m_{u}=m_{d}$) as well as learning different ones ($m_{u}\neq m_{d}$).

### 3.2 Sentiment Control

Following prior work (Rafailov et al., ; Amini et al., ; Zeng et al., ),
we use sentiment as a proxy for preference and align models to generate positive movie reviews.
For the SFT model, we use GPT2-Large (Radford et al., ) trained on
the IMDB dataset (Maas et al., ).(^2^22[https://huggingface.co/insub/gpt2-large-imdb-fine-tuned](https://huggingface.co/insub/gpt2-large-imdb-fine-tuned))
To train PO, preference data is generated by sampling two completions per review prefix from the SFT model.
Then, we use a sentiment classifier(^3^33[https://huggingface.co/siebert/sentiment-roberta-large-english](https://huggingface.co/siebert/sentiment-roberta-large-english)) as a ground-truth reward model and set chosen ($y_{c}$) and rejected ($y_{r}$) responses
such that $\text{score}(y_{c})>\text{score}(y_{r})$, where $\text{score(y)}=p(y|\text{positive})$ or
$1-p(y|\text{negative})$ if $y$ is classified as positive or negative, respectively.

Figure: Figure 2: Pareto frontier of expected reward and response-level KL divergence w.r.t. the reference model, for a sentiment control scenario over the IMDB dataset. Solid lines estimate the frontier for each system, and points represent hyper-parameter variations.
Refer to caption: x1.png

Reward and KL Divergence Trade-off.
We start our analysis by investigating the trade-off between ground-truth reward and response-level KL divergence by estimating their Pareto frontier. For all policies, we train using $\beta=\{0.01,0.1,0.2,...,1,2,3,4,5,10,20\}$ and
for SimPO, $\gamma$ in $\{0.3,1\}$.
For each policy variation, we generate one response per prompt in the test set using multinomial sampling, every 100 training steps, and report the the ground-truth reward and the average response-level KL divergence, averaged over samples.

The following insights can be gathered from the frontier, showcased in Figure 2.
We observe that DPOP greatly restricts KL divergence and maintains a lower reward level than other systems, whereas SimPO and TDPO v1 allows for larger KL divergence and improved reward.
In contrast, TDPOv2 allows slightly larger KL divergence than v1 but it does not reach higher rewards.
Among our proposed systems, MaPO notably dominates the frontier at moderate divergence levels, reaching higher rewards than DPO and comparable to SimPO and TDPOv1.
On the other hand, SparsePO variants allow a much larger effective KL divergence range, with higher concentration of system points at high KL values than any baseline.
Regarding rewards, although the independent mask setup ($m_{u}\neq m_{d}$) does reach reward levels of MaPO, the common mask setup ($m_{u}=m_{d}$) seems to trade off divergence range for a slight decrease in reward.
These results demonstrate that the proposed masking strategies are effective at balancing expected ground-truth reward and response-level KL divergence.

Sparsity and Token-level KL divergence.
Next, we analyze the trade-off between mask sparsity and token-level KL divergence throughout training,
in the independent mask setup of SparsePO.
Figure 3 shows results for chosen responses from systems trained at different values of $\beta$.(^4^44See Figure 7 in Appendix C for similar plots over rejected responses.)
Firstly, we note that sparsity in the reward mask ($m_{u}$)
always starts high ($80\%$), increasing slightly and then steadily decreasing until the end of training, reaching as down as $20\%$.
Such decrease is controlled by increasing $\beta$ until $0.8$, after which the trend is inverted.
We hypothesize that the reward mask first learns to identify the tokens most informative for sentiment control,
and increasingly expands this token set as training proceeds at a rate controllable by $\beta$.
This insight adds to previous findings (Yang et al., ) stating that PO-trained models can learn to identify highly rewardable tokens.

Regarding the divergence mask, we find that increasingly higher values of $\beta$ induce higher levels of sparsity in $m_{d}$,
restricting the amount of tokens allowed to diverge in a sequence, which translates to lower token-level KL divergence throughout training.
However, for sufficiently low values of $\beta$, sparsity can be kept below $20\%$.

In summary, we find that low values of $\beta$ induce scenarios where reward sparsity is high and divergence sparsity is low, meaning that the loss is dominated by the masked divergence term, $\delta(x,y_{c},y_{r})$.
Conversely, high values of $\beta$ induce high sparsity on both masks, hindering learning significantly.
However, we do observe that more balanced sparsity levels in both masks can be induced with mid-range values of $\beta$.

Figure: Figure 3: Sparsity levels in the reward mask ($m_{u}$, left) and the token-level KL divergence mask ($m_{d}$, middle), as well as token-level KL divergence of chosen responses during training (over IMDB), for increasing values of $\beta$.
Refer to caption: x2.png

Figure: (a) Chosen response rewards.
Refer to caption: extracted/5910009/tdpov2_chosen_rewards.png

Qualitative Analysis.
Finally, we perform qualitative analysis on the learned masks by observing their token-level values on example sentences.
Similarly to Figure 1, we calculate token-level rewards as the log ratio of response probabilities between policy and reference models. Token-level KL divergence is calculated as the token-level KL between policy and reference.
We show the values of reward and KL divergence after the mask application in a common mask setup($m_{u}=m_{d}\rightarrow$ common) and on independent setup ($m_{u}\neq m_{d}\rightarrow$ indp).
We also compare with the TDPO baseline as the closest method to ours. Technically, when $m_{u}=m_{d}=1$ our objective becomes equivalent to TDPO, hence we can check the influence of the proposed masks on the target objective.
Figure 4(a) illustrates that a common mask has less sparsity compared to independent, highlighting a larger set of tokens. Comparing directly reward maps with TDPO we see that that independent mask is weighting only subsequences that express a certain polarity (watch it again), while TDPO gives a weight to all tokens in the sequence. The same stands for common masks while being slightly noisier in the tokens they cover.
Looking at KL divergence maps in Figure 4(b), lower values indicate minor to no divergence from the reference model. TDPO is stricter in KL control, forcing the majority of tokens to be close to the reference model, while common and sparse masks allow more diversity with higher values on particular tokens, possibly easing diversity. Heatmaps for the rejected response can be found in Figure 9.

### 3.3 Helpfulness & Harmlessness Control

Here, we investigate the effectiveness of our approach in aligning models to generate
helpful and harmless responses in dialogue.
We employ the Anthropic HH dataset (Bai et al., ), consisting of open-ended multi-turn dialogues in which humans ask a chat assistant for help, advice, or to perform a task.
We train Pythia 1.4B (Biderman et al., )
using the chosen completions for SFT training and the preference dataset for PO over the resulting reference model.

In Table 1, we report performance in reasoning and instruction following tasks over HuggingFace’s OpenLLM Leaderboard v2.(^5^55[https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard))For metric calculation, we use the LM Evaluation Harness framework (Gao et al., ).

Alignment, Reasoning and Verifiable Instruction Following.
In terms of average score, SFT performs better than all systems, indicating a sharp trade-off between alignment objective and task performance, regardless of the PO strategy.
This could indicate that by making a model more helpful and harmless, we sacrifice some reasoning capabilities (Luo et al., ).
Nevertheless, our proposed alignment strategies, MaPO and SparsePO --both at common and independent mask setups-- demonstrate their effectiveness at balancing alignment goals and reasoning, being the best among PO strategies.

In terms of specific reasoning and task type, the following can be noted.
Firstly, our mask strategies are effective for certain types of reasoning.
Although mathematical reasoning (MATH) poses a challenge to all systems,
MaPO outperforms all baselines including SFT, followed by SparsePO$[m_{u}\neq m_{d}]$.
Similarly, SparsePO$[m_{u}\neq m_{d}]$ performs best at BBH, followed by MaPO, indicating a better handling of factual and world knowledge as well as algorithmic reasoning.
Multi-step soft reasoning tasks (MuSR) are best handled by SparsePO$[m_{u}=m_{d}]$, followed by TDPOv2.
However, tasks that require extensive knowledge (GPQA and MMLU-pro) pose a challenge to all systems, and our masking strategies in particular.
Similarly, tasks based on verifiable instructions (IFEval), both instruction and prompt based, exhibit the starkest trade-off between alignment and task performance, given the sharp decrease in metric scores after preference optimization.
Still, SparsePO and MaPO outperform all other PO strategies, trailing second only to SFT.

**Table 1: Performance of Pythia 1.4B models on Open LLM Leaderboard 2 after PO with Helpfulness & Harmlessness as proxy for human preference. Best number across PO methods are bolded.**
| Methods | BBH | MATH | GPQA | MuSR | MMLU | IFEval | Avg |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | pro | Inst. | Prom. |  |
| SFT | 2.87 | 0.30 | 0.78 | 4.02 | 1.71 | 25.90 | 14.97 | 7.22 |
| DPO | 2.64 | 0.60 | 0.00 | 3.77 | 1.19 | 21.46 | 10.54 | 5.74 |
| TDPO v1 | 3.01 | 0.53 | 0.00 | 4.30 | 1.50 | 20.62 | 9.98 | 5.71 |
| TDPO v2 | 2.65 | 0.23 | 0.00 | 5.87 | 1.68 | 18.47 | 8.32 | 5.32 |
| SimPO | 2.10 | 0.00 | 1.12 | 4.36 | 1.41 | 19.90 | 9.24 | 5.45 |
| DPOP | 2.71 | 0.68 | 1.57 | 3.85 | 1.43 | 20.02 | 9.06 | 5.62 |
| MaPO | 3.60 | 0.91 | 0.00 | 3.94 | 1.33 | 22.78 | 12.57 | 6.45 |
| SparsePO$[m_{u}=m_{d}]$ | 3.24 | 0.23 | 0.00 | 6.67 | 1.25 | 22.78 | 12.38 | 6.65 |
| SparsePO$[m_{u}\neq m_{d}]$ | 4.10 | 0.76 | 0.00 | 3.45 | 1.42 | 22.78 | 11.28 | 6.25 |

### 3.4 Summary Quality Control

In this task, we employ overall summary quality as proxy for human preference, which includes quality aspects such as information coverage, faithfulness, and coherence.
We use the Reddit TL;DR dataset (Völske et al., ) and its preference annotations (Stiennon et al., ) to
fine-tune a GPTJ-6B (Wang & Komatsuzaki, ) SFT model(^6^66[https://huggingface.co/CarperAI/openai_summarize_tldr_sft](https://huggingface.co/CarperAI/openai_summarize_tldr_sft)) using QLoRA (Dettmers et al., ).
Here we only analyze representative baselines from sequence and token-level preference modeling (DPO, TDPO v1 and v2) against MaPO and SparsePO in common mask setup.

For evaluation, we take 120 prompts from the test set and sample 5 completions using nucleus sampling ($p=0.95$) and temperatures $T=\{0.25,0.50,0.75,1.0\}$.
Regarding automatic metrics, we report ROUGE-L F1 (Lin & Hovy, ) and BERTScore F1 (Zhang et al., ) for lexical and semantic relevance, respectively;
self-BLEU (Zhu et al., ) for lexical diversity;
and EDNA (Narayan et al., ), a metric quantifying diversity and faithfulness by combining
document-summary entailment (Laban et al., ) and self-entailment.

Alignment, Diversity, and Faithfulness.
We investigate how our method balances alignment accuracy --as measured by summary relevancy--, generation diversity, and faithfulness.
Figure 5 presents metric scores across temperature values.
Notably, SparsePO achieves a better trade-off between relevancy and diversity than the baselines, obtaining higher ROUGE-L and BERTScore values for comparable Self-BLEU scores.
However, EDNA scores indicate that these balance comes at the expense of entailment-based
faithfulness and diversity.
For all metrics, the performance gap reduces as temperature increases, and all models obtain comparable results at temperature $1.0$.

Figure: Figure 5: Summary relevancy (ROUGE-L, BERTScore; $\uparrow$), lexical diversity (Self-BLEU, $\downarrow$), and entailment-based diversity and faithfulness (EDNA, $\uparrow$), over the test set of TL;DR.
Refer to caption: x3.png

### 3.5 Text-to-Code Generation

Finally, we perform preference optimization for the task of text-to-code generation, using a simple preference dataset created from Python programming problems from Gee et al. ().
In this experiment, we aim to optimize for correctness, i.e., a chosen program is an executionable one that passes all accompanied unit-tests and a rejected program is one with the opposite behavior.

The MBPP dataset (Austin et al., ) is employed, which consists of 384 train, 90 validation and 500 test programs.
We use StarCoder-1B (Li et al., ) to sample 100 solutions for each problem in train and validation with multinomial sampling.
After sampling and testing the generated programs, we end up with 183 prompts with at least two passing and one failed solution for the training set and 40 for the validation set.
Using the resulting data, we train StarCoder-1B with different preference optimization algorithms.
We select randomly different pass-fail solutions for each prompt at every epoch. The checkpoint that has the lowest validation set loss is chosen for testing.
Performance is measured in terms of functional correctness on MBPP and HumanEval (Austin et al., ), sampling 100 solutions with temperature $0.6$ and $p=0.95$ in Table 2.

Overall, DPO shows the strongest performance across the board on HumanEval for all pass$@k$ setups, while all methods manage to improve over the baseline SFT model.
Our proposed models tend to perform on par with other PO methods although worse on pass@100. On MBPP though, SparsePO shows gains over pass@100, offering a $+2\%$ improvement compared to DPO, with a slight decay in the remaining metrics. The discrepancy between HumanEval and MBPP could be attributed to the MBPP being the in-domain PO data.
These results indicate that although SparsePO is weighting more tokens as important for preference, in the code domain and in particular code execution, this requirement cannot be easily satisfied.
Intuitively, it is challenging to identify which particular tokens are more responsible for a program executing correctly, as indicated by low mask sparsity levels (shown below).
Arguably, the entire structure of the code needs to adhere to certain rules for syntactic and semantic correctness.
Despite the above, increased pass@100 performance could be an indication of more diverse generation.

**Table 2: Pass@$k$ results for text-to-code generation using StarCoder-1B.**
|  | HumanEval | MBPP |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | pass@1 | pass@10 | pass@100 | pass@1 | pass@10 | pass@100 |
| StarCoder-1B | 12.22 | 24.69 | 38.41 | 17.83 | 39.94 | 59.60 |
| DPO | 14.61 | 28.42 | 46.34 | 21.36 | 44.71 | 62.40 |
| TDPO-v1 | 14.46 | 27.42 | 46.34 | 21.58 | 44.48 | 61.60 |
| TDPO-v2 | 13.30 | 26.06 | 45.73 | 19.93 | 42.51 | 62.00 |
| SimPO | 14.55 | 27.74 | 45.73 | 22.89 | 43.63 | 59.20 |
| MAPO | 14.12 | 27.30 | 42.07 | 20.93 | 43.63 | 62.20 |
| SparsePO$[m_{u}=m_{d}]$ | 14.15 | 27.32 | 42.68 | 20.92 | 44.25 | 64.80 |
| SparsePO$[m_{u}\neq m_{d}]$ | 14.39 | 28.29 | 44.51 | 19.81 | 43.71 | 62.00 |

Sparsity and Token-level KL divergence.
Similarly to sentiment control, we also report sparsity values as a function of training steps for models trained with different values of $\beta$. Figure 6 shows sparsity and token-level KL divergence for chosen responses (Figure 8 in the Appendix for rejected responses). Higher values of $\beta$ do offer significant KL control, resulting into lower KL. Sparsity is much lower for reward masks and higher for KL masks, with both being relatively stable within a small range of values ($\pm$ 4-6 points).
Intuitively, since code is stricter in syntax compared to natural language, several tokens should be considered important for correct execution of a program (e.g. correct signature, indentation, return values, etc), supporting our aforementioned argument. On the other hand, larger KL divergence can offer flexibility in the surface form of code variable names.

Figure: Figure 6: Sparsity levels in the reward mask ($m_{u}$, left), the token-level KL divergence mask ($m_{d}$, middle), and token-level divergence of chosen responses during training MBPP), for increasing $\beta$.
Refer to caption: x4.png

## 4 Related Work

Since the introduction of DPO, several methods have been developed to mitigate the various shortcomings of the method, mostly by introducing further constrains to the loss function.
Identity Preference Optimization (Gheshlaghi Azar et al., , IPO) was proposed to primarily tackle overfitting, that does not rely on the Bradley-Terry modulation assumption.
Ethayarajh et al. () introduced KTO, that takes advantage of that Kahneman-Tversky model of human utility. The method drops the requirement for preference pairs and is dependent only on a binary signal of whether a response is acceptable or not.
To control response length and dismiss the need for a reference model, SimPO (Meng et al., ) uses the average log probability of the sequence (instead of the sum) while also requiring the difference between responses to be at least equal to a margin.
Another method that does not require a reference model or prior supervised fine-tuning, is ORPO (Hong et al., ), that optimizes the odds ratio together with cross-entropy.
On a similar vein, Amini et al. () argues that not all preference pairs are considered equal, requiring the preferred responses to have a likelihood larger than an offset value from the dispreferred ones, based on the score assigned to each response from an external reward model.
Other methods that incorporate margins between probability differences include DPO-Positive (Pal et al., ), where the log probability of the preferred response for the policy needs to be higher than that of the reference model. The method is particularly effective when the edit distance between responses is low, e.g in math data.
Wu et al. () specifically aimed at a dynamic optimization of the $\beta$ value for each batch, proposing $\beta$-DPO.

Closer to our approach, there is a family of methods that focus on token-level rather than sequence-level optimization.
In TDPO (Zeng et al., ), the sequence-level DPO objective is converted into token-level, which results in the KL divergence to act as a regularization term, optimized together with the original objective. The new loss leads to more controllable KL values throughout the course of training.
Inverse-Q*(Xia et al., ) optimizes the same objective as PPO assigning token-level reward feedback via an estimated policy.
Similarly, Token-level Continuous Rewards (Yoon et al., , TLCR) incorporate a discriminator trained to distinguish positive and negative tokens (obtained from GPT-4 judgments). The confidence of the discriminator is used to assign continuous rewards to each token considering the context.
Similarly to our motivation, in Selective PO (Yang et al., , SePO), not all tokens are considered equal. An oracle model is trained first to identify which tokens are important in chosen and rejected responses (based on their reward values). These tokens are then used to train DPO again, while the rest are zeroed out.
In contrast to the above methods, we aim for maximum flexibility. Our approach does not require an external LLM to model rewards and our proposed masks are learned on the fly, effectively assigning higher rewards to tokens that are important to the target preference. In addition, SparsePO induces the necessary sparsity in the masks automatically with a single stage of training.

## 5 Discussion

Based on the controlled experiments we conducted in the previous section, here we briefly discuss our overall findings.
Firstly, based on the sentiment control analysis, SparsePO allows larger KL divergence at little to no cost in expected ground-truth reward.
The $\beta$ value is able to control sparsity in both masks,
across domains, with values between $0.6$ to $4$ leading to mid-range sparsity levels.
Depending on the domain and target preference proxy, we found that higher sparsity was present in sentiment control, highlighting a certain triviality of the task as the SFT model seems able to already identify words that are important for the target preference.
On the other end, for code generation and summarization, lower sparsity between $0.2$ and $0.4$ seemed best in terms of alignment accuracy as executability and summary correctness are less well-defined preference proxies.
For helpfulness control, optimal sparsity was found instead between $0.6$ and $0.8$, possibly as existence of toxic terms immediately renders response dispreferred.

From our analysis over DPO, TDPO and their variants, it is important to note that,
although restricting divergence at the response or token-level proves effective at maintaining the model in-domain, this does not guarantee better ground-truth rewards or better downstream task performance.
For cases in which the preference proxy is complex, such as ‘helpfulness’, ‘summary quality’ or ‘executability’, this plain control can even hinder performance.
In contrast, we devise a training procedure in which a model can learn to enhance or suppress the reward and KL divergence for each token independently.
Our qualitative analysis shows that indeed for trivial tasks tokens important towards the preference get high rewards and low KL divergence, meaning they need to be close to the reference predictions to maintain preference.

## 6 Conclusion

We introduced Sparse Token-level Preference Optimization (SparsePO), a novel LM alignment strategy that learns to weight the reward and KL divergence for each particular token in a response during PO training.
We proposed two masking strategies, obtaining model activation-based masks from the reference model and learning mask representations either commonly for both reward and divergence terms or independently.
By allowing masks to be learned along with preference,
we observed that they converged to a non-trivial level of sparsity which can be controlled with well-studied hyper-parameters in preference optimization, while being dependent on target preference proxy.
Extensive experiments across several tasks and domains, reveal that
our method consistently outperforms strong baselines that model preference at the response and token-level, while assigning higher rewards and lower KL values to tokens that are important for inferring target preference.
SparsePO can be easily extended to use other masking strategies and can be combined with other PO variations.

## Appendix A Mathematical Derivations

### A.1 Obtaining the Optimal Policy

In order to get the optimal policy, we take advantage of $A(y^{t}|x,y^{<t})\equiv Q(y^{t}|x,y^{<t})-V(x,y^{<t})$ and solve the following objective that includes our introduced mask $m(y^{<t})$.
In the following equations, $\pi$ refers always to next-token distribution $\pi(\cdot|x,y^{<t})$,
and we oftentimes omit $(y^{t}|x,y^{<t})$ for simplicity.

$$ $\displaystyle J_{\pi}$ $\displaystyle=\max_{\pi}\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\left[A_{\pi_{ \text{ref}}}(y^{t}|x,y^{<t})\right]-\beta\;m(y^{<t})\;D_{\mathrm{KL}}[\pi( \cdot|x,y^{<t})||\pi_{\text{ref}}(\cdot|x,y^{<t})]$ $\displaystyle=\max_{\pi}\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\left(\left(Q_ {\pi_{\text{ref}}}(y^{t}|x,y^{<t})-V_{\pi_{\text{ref}}}(x,y^{<t})\right)+\beta \;{m(y^{<t})}\;\log\left(\frac{\pi_{\text{ref}}(y^{t}|x,y^{<t})}{\pi(y^{t}|x,y ^{<t})}\right)\right)$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\left (\log e^{\frac{1}{\beta}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})}-\frac{1}{\beta}V _{\pi_{\text{ref}}}(x,y^{<t})+\log\left(\frac{\pi_{\text{ref}}(y^{t}|x,y^{<t}) }{\pi(y^{t}|x,y^{<t})}\right)^{{m(y^{<t})}}\right)$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\log \left(\frac{\pi_{\text{ref}}^{{m(y^{<t})}}(y^{t}|x,y^{<t})\exp\left(\frac{1}{ \beta}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})\right)}{\pi^{{m(y^{<t})}}(y^{t}|x,y ^{<t})}\right)-\frac{1}{\beta}V_{\pi_{\text{ref}}}(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\log \left(\frac{\frac{Z(x,y^{<t})}{Z(x,y^{<t})}\pi_{\text{ref}}^{{m(y^{<t})}}\exp \left(\frac{1}{\beta}Q_{\pi_{\text{ref}}}\right)}{\pi^{{m(y^{<t})}}}\right)- \frac{1}{\beta}V_{\pi_{\text{ref}}}$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\log \left(\frac{\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}^{{m(y^{<t})}}\exp\left(\frac {1}{\beta}Q_{\pi_{\text{ref}}}\right)}{\pi^{{m(y^{<t})}}}\right)-\frac{1}{ \beta}V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\log \left(\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}^{{m(y^{<t})}}\exp\left(\frac{1}{ \beta}Q_{\pi_{\text{ref}}}\right)\right)-\log\pi^{{m(y^{<t})}}-\frac{1}{\beta} V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}\frac {{m(y^{<t})}}{{m(y^{<t})}}\log\left(\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}^{{m( y^{<t})}}\exp\left(\frac{1}{\beta}Q_{\pi_{\text{ref}}}\right)\right)-{m(y^{<t} )}\log\pi-\frac{1}{\beta}V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}{m(y^ {<t})}\log\left(\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}^{{m(y^{<t})}}\exp\left( \frac{1}{\beta}Q_{\pi_{\text{ref}}}\right)\right)^{\frac{1}{{m(y^{<t})}}}-{m(y ^{<t})}\log\pi-\frac{1}{\beta}V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}{m(y^ {<t})}\left(\log\left(\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}^{{m(y^{<t})}}\exp \left(\frac{1}{\beta}Q_{\pi_{\text{ref}}}\right)\right)^{\frac{1}{{m(y^{<t})}} }-\log\pi\right)-\frac{1}{\beta}V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}{m(y^ {<t})}\left(\log\left(\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}\exp\left(\frac{1}{ \beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}}}\right)\right)-\log\pi\right)-\frac{1}{ \beta}V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;\beta\;\mathbb{E}_{x,y^{<t}\sim D,y^{t}\sim\pi}{m(y^ {<t})}\log\left(\frac{\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}\exp\left(\frac{1}{ \beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}}}\right)}{\pi}\right)-\frac{1}{\beta}V_{ \pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\max_{\pi}\;-\beta\;D_{\mathrm{KL}}\left({\pi}{\|}{\frac{1}{Z(x, y^{<t})}\pi_{\text{ref}}\exp\left(\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{\text{ ref}}}\right)}\right)-\frac{1}{\beta}V_{\pi_{\text{ref}}}+\log Z(x,y^{<t})$ $\displaystyle=\min_{\pi}\beta\;D_{\mathrm{KL}}\left({\pi}{\|}{\frac{1}{Z(x,y^{ <t})}\pi_{\text{ref}}\exp\left(\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}} }\right)}\right)+\frac{1}{\beta}V_{\pi_{\text{ref}}}-\log Z(x,y^{<t})$ (9) $$

Where the partition function is given by:

$$ $\displaystyle Z(x,y^{<t})$ $\displaystyle=\mathbb{E}_{y^{t}\sim\pi_{\text{ref}}}\;\pi_{\text{ref}}(y^{t}|x ,y^{<t})\;\exp\left(\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}}}(y^{t}|x,y ^{<t})\right)$ $\displaystyle=\sum_{y^{<t}}\pi_{\text{ref}}(y^{t}|x,y^{<t})\;\exp\left(\frac{1 }{\beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})\right).$ (10) $$

The objective in Equation A.1 can be minimized if the KL term becomes zero (as $Z$ and $V_{\pi_{\text{ref}}}$ are not dependent on $\pi$), which effectively equals to the optimal policy becoming

$$ $\pi^{*}(y^{t}|x,y^{<t})=\frac{1}{Z(x,y^{<t})}\pi_{\text{ref}}(y^{t}|x,y^{<t}) \exp\left(\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t}) \right).$ (11) $$

### A.2 Deriving the SparsePO Objective from the Bradley-Terry Equivalence

The equivalence of Bradley-Terry with the Regret Preference Model, its equivalent on the token-level, has been previously proven in Zeng et al. () as the probability of preferring a chosen response $y_{c}$ over a rejected response $y_{r}$,

$$ $P_{\text{BT}}(y_{c}>y_{r}|x)=\sigma\left(\sum_{t=1}^{T_{1}}A_{\pi}(y_{c}^{t}|x ,y_{c}^{<t})-\sum_{t=1}^{T_{2}}A_{\pi}(y_{r}^{t}|x,y_{r}^{<t})\right)$ (12) $$

Replacing $A_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})\equiv Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t}
)-V_{\pi_{\text{ref}}}(x,y^{<t})$ in Equation 12 and considering that $V_{\pi_{\text{ref}}}(x,y^{<t})=\mathbb{E}_{\pi_{\text{ref}}}[Q_{\pi_{\text{ref
}}}(y^{t}|x,y^{<t})]$ we have

$$ $\displaystyle\sum_{t=1}^{T}A_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})$ $\displaystyle=\sum_{t=1}^{T}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})-V_{\pi_{\text {ref}}}(x,y^{<t})$ $\displaystyle=\sum_{t=1}^{T}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})-\mathbb{E}_{y ^{t}\sim\pi_{\text{ref}}}[Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})]$ (13) $$

Adding logarithms in front of each part of Equation 11 and solving for $Q_{\pi_{\text{ref}}}$, we get

$$ $\displaystyle\log\pi^{*}(y^{t}|x,y^{<t})=\log\left(\frac{1}{Z(x,y^{<t})}\pi_{ \text{ref}}(y^{t}|x,y^{<t})\exp\left(\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{\text {ref}}}(y^{t}|x,y^{<t})\right)\right)$ $\displaystyle\log\pi^{*}(y^{t}|x,y^{<t})=\log\left(\frac{1}{Z(x,y^{<t})}\right )+\log\pi_{\text{ref}}(y^{t}|x,y^{<t})+\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{ \text{ref}}}(y^{t}|x,y^{<t})$ $\displaystyle\log\pi^{*}(y^{t}|x,y^{<t})-\log\pi_{\text{ref}}(y^{t}|x,y^{<t})= -\log Z(x,y^{<t})+\frac{1}{\beta\;{m(y^{<t})}}Q_{\pi_{\text{ref}}}(y^{t}|x,y^{ <t})$ $\displaystyle Q_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})=\beta\;{m(y^{<t})}\log\frac {\pi^{*}(y^{t}|x,y^{<t})}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}+\beta\;{m(y^{<t})} \log Z(x,y^{<t})$ (14) $$

Now, leveraging Equation 14, Equation A.2 becomes

$$ $\displaystyle\sum_{t=1}^{T}A_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})$ $\displaystyle=\sum_{t=1}^{T}\beta\;{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x,y^{<t} )}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}+\beta\;{m(y^{<t})}\log Z(x,y^{<t})$ $\displaystyle-\mathbb{E}_{y^{t}\sim\pi_{\text{ref}}}[\beta\;{m(y^{<t})}\log \frac{\pi^{*}(y^{t}|x,y^{<t})}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}+\beta\;{m(y^{ <t})}\log Z(x,y^{<t})]$ $\displaystyle=\sum_{t=1}^{T}\beta\;{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x,y^{<t} )}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}+\beta\;{m(y^{<t})}\log Z(x,y^{<t})$ $\displaystyle-\mathbb{E}_{y^{t}\sim\pi_{\text{ref}}}[\beta\;{m(y^{<t})}\log \frac{\pi^{*}(y^{t}|x,y^{<t})}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}]-\mathbb{E}_{ y^{t}\sim\pi_{\text{ref}}}[\beta\;{m(y^{<t})}\log Z(x,y^{<t})]$ (15) $$

Since $m(y^{<t})$ depends only on the previously seen tokens (and not the current one), we can say that $\mathbb{E}_{y^{t}\sim\pi_{\text{ref}}}\;[\beta\;m(y^{<t})\;\log Z(x,y^{<t})]=
\beta\;m(y^{<t})\;\mathbb{E}_{y^{t}\sim\pi_{\text{ref}}}[\log Z(x,y^{<t})]=
\beta\;m(y^{<t})\;\log Z(x,y^{<t})$.
Replacing the above to Equation 15,

$$ $\displaystyle\sum_{t=1}^{T}A_{\pi_{\text{ref}}}(y^{t}|x,y^{<t})$ $\displaystyle=\sum_{t=1}^{T}\left(\beta\;{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x, y^{<t})}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}-\mathbb{E}_{y^{t}\sim\pi_{\text{ref }}}\left[\beta\;{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x,y^{<t})}{\pi_{\text{ref}} (y^{t}|x,y^{<t})}\right]\right)$ $\displaystyle=\sum_{t=1}^{T}\left(\beta\;{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x, y^{<t})}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}-\beta\;{m(y^{<t})}\;D_{\mathrm{KL}} [\pi^{*}(\cdot|x,y^{<t})\|\pi_{\text{ref}}(\cdot|x,y^{<t})]\right)$ $\displaystyle=\sum_{t=1}^{T}\beta\;{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x,y^{<t} )}{\pi_{\text{ref}}(y^{t}|x,y^{<t})}-\sum_{t=1}^{T}\beta\;{m(y^{<t})}\;D_{ \mathrm{KL}}[\pi^{*}(\cdot|x,y^{<t})\|\pi_{\text{ref}}(\cdot|x,y^{<t})]$ $\displaystyle=\beta\sum_{t=1}^{T}{m(y^{<t})}\log\frac{\pi^{*}(y^{t}|x,y^{<t})} {\pi_{\text{ref}}(y^{t}|x,y^{<t})}-\beta\sum_{t=1}^{T}{m(y^{<t})}\;D_{\mathrm{ KL}}[\pi^{*}(\cdot|x,y^{<t})\|\pi_{\text{ref}}(\cdot|x,y^{<t})]$ (16) $$

Finally, replacing the result of Equation A.2 that into Equation 12

$$ $\displaystyle P_{\text{BT}}(y_{c}>y_{r}|x)=$ (17) $\displaystyle\sigma\Big{(}\beta\sum_{t=1}^{T_{1}}{m(y_{c}^{<t})}\log\frac{\pi^ {*}(y^{t}_{c}|x,y^{<t}_{c})}{\pi_{\text{ref}}(y^{t}_{c}|x,y^{<t}_{c})}-\beta \sum_{t=1}^{T_{1}}{m(y_{c}^{<t})}\;D_{\mathrm{KL}}[\pi^{*}(\cdot|x,y^{<t}_{c}) \|\pi_{\text{ref}}(\cdot|x,y^{<t}_{c}))$ $\displaystyle\Big{.}-\beta\sum_{t=1}^{T_{2}}{m(y_{r}^{<t})}\log\frac{\pi^{*}(y ^{t}_{r}|x,y^{<t}_{r})}{\pi_{\text{ref}}(y^{t}_{r}|x,y^{<t}_{r})}+\beta\sum_{t =1}^{T_{2}}{m(y_{r}^{<t})}\;D_{\mathrm{KL}}[\pi^{*}(\cdot|x,y^{<t}_{r})\|\pi_{ \text{ref}}(\cdot|x,y^{<t}_{r})]\Big{)}$ $$

Where we define,

$$ $\displaystyle u(x,y_{c},y_{r})$ $\displaystyle=\beta\sum_{t=1}^{T_{1}}{m_{u}(y_{c}^{<t})}\log\frac{\pi^{*}(y^{t }_{c}|x,y^{<t}_{c})}{\pi_{\text{ref}}(y^{t}_{c}|x,y^{<t}_{c})}-\beta\sum_{t=1} ^{T_{2}}{m_{u}(y_{r}^{<t})}\log\frac{\pi^{*}(y^{t}_{r}|x,y^{<t}_{r})}{\pi_{ \text{ref}}(y^{t}_{r}|x,y^{<t}_{r})}$ (18) $$

$$ $\displaystyle\delta(x,y_{c},y_{r})$ $\displaystyle=\beta\sum_{t=1}^{T_{1}}{m_{d}(y_{c}^{<t})}\;D_{\mathrm{KL}}[\pi^ {*}(\cdot|x,y^{<t}_{c})\|\pi_{\text{ref}}(\cdot|x,y^{<t}_{c})]$ (19) $\displaystyle-\beta\sum_{t=1}^{T_{2}}{m_{d}(y_{r}^{<t})}\;D_{\mathrm{KL}}[\pi^ {*}(\cdot|x,y^{<t}_{r})\|\pi_{\text{ref}}(\cdot|x,y^{<t}_{r})]$ $$

Resulting in

$$ $\displaystyle p_{BT}(y_{c}>y_{r}|x)=\sigma\left(u(x,y_{c},y_{r})-\delta(x,y_{c },y_{r})\right)$ (20) $$

Formulating the maximum likelihood objective given the probability of human preference data in terms of optimal policy in Equation 20, the loss function becomes

$$ $\mathcal{L}_{SparsePO}=-\mathbb{E}_{(x,y_{c},y_{r})\sim D}[\log\sigma\left(u(x ,y_{c},y_{r})-\delta(x,y_{c},y_{r})\right)]$ (21) $$

## Appendix B Details on Experimental Setup

In this appendix, we provide further details on the experimental setup.
All experiments
used AdamW optimizer (Kingma & Ba, ).

### B.1 Release

Uppon acceptance, the code will be released in the following repository:
[https://github.com/huawei-noah/noah-research/tree/master/NLP/sparse_po](https://github.com/huawei-noah/noah-research/tree/master/NLP/sparse_po)

### B.2 Sentiment Control

Dataset.
We use the IMDB dataset preprocessed for preference optimization by Amini et al. (), which uses
prefixes of length 5-8 tokens as prompts.

Training and Optimization.
All models are trained over three epochs with an effective batch size of $64$.
For TDPO, we set $\alpha=0.7$, as it was reported as best for IMDB in Zeng et al. ().
For DPOP, we set $\lambda=50$ as reported by Pal et al. ().

### B.3 Summary Quality Control

Dataset.
For preference optimization, we use the TL;DR feedback dataset collected by Stiennon et al. (), comprising of two subsets, one with pairwise comparison and the other with the individually rated
summaries.
Following Amini et al. (), we binarize the single-summary subset by selecting the summary with highest and lowest overall Likert score as the chosen and rejected response,respectively.
In order to mitigate the compounding effect of summary length, we filtered out training instances with chosen and rejected responses with a length difference greater than 100 words.
From these resulting filtered dataset, We uniformly sample 20k and 4k preference instances from each subset to form a training and test set of 40k and 8k instances, respectively.

Training and Optimization.
All models are trained using QLora with parameters rank $r=16$, $\alpha=16$, and dropout $0.05$, over 4bit quantized model weights.
Training is done for one epoch with an effective batch size of $256$ and learning rate of $1e^{-5}$.
We set $\beta=0.5$ for all PO baselines and MaPO; $\alpha=0.5$ for TDPO v1 and v2;
and a $\beta=0.1$ and weight decay of $0.01$ over mask weights for SparsePO.

Evaluation.
Statistical significance at the system level is tested pairwise using Bootstrap resampling (Davison & Hinkley, ) with a $95\%$ confidence interval.
For ROUGE, we report results using stemming; for BERTScore, we use RoBERTa large (Liu et al., ) as underlying model with sentence-level IDF importance weighting, for which the scores were calculated over the training set. EDNA scores we calculated via the SummaC framework.(^7^77[https://github.com/tingofurro/summac](https://github.com/tingofurro/summac))

### B.4 Helpfulness & Harmlessness Control

Dataset.
We use the Anthropic HH dataset available in HuggingFace.(^8^88[https://huggingface.co/datasets/Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf))

Training and Optimization.
The reference model is trained for one epoch over chosen responses with a learning rate of $1e^{-5}$ and an effective batch size of $1024$.
Preference policy models are trained for three epochs at full precision with an effective batch size of $128$, learning rate of $1e^{-6}$, and, otherwise specified, $\beta=0.1$.
For TDPO v1 and v2, we set $\alpha=0.5$ as it performed better in preliminary experiments.
Similarly, we set $\beta=2.5$ and $\gamma=0.3$ for SimPO.
For SparsePO, we set a learning rate of $5e^{-7}$, mask weight decay of $0.01$, and L1 normalization parameter of $0.001$ for both reward and KL masks.

Evaluation.
For OpenLLM leaderboard evaluation, we employ EleutherAI Evaluation Harness library (Gao et al., ) and report scores normalized across tasks, as recommended by the leaderboard authors.(^9^99[https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization))
In this way, individual task scores are reported in the same 0-100 scale, and final average scores are not biased toward one single task.

### B.5 Text-to-Code Generation

Dataset.
The MBPP dataset (Austin et al., )(^10^1010[https://huggingface.co/datasets/google-research-datasets/mbpp](https://huggingface.co/datasets/google-research-datasets/mbpp)) is employed, which consists of 384 train, 90 validation and 500 test programs. We preserve the test set for final evaluation and use the remaining sets for PO training.

Training and Optimization.
We train StarCoder-1B (Li et al., )(^11^1111[https://huggingface.co/loubnabnl/starcoder-1b](https://huggingface.co/loubnabnl/starcoder-1b)) for $30$ epochs with a learning rate of $5e^{-7}$, a warmup of $10\%$ of the total training steps, linear learning rate decay and an effective batch size of $32$.

Evaluation.
For evaluation we employ the BigCode-evaluation-harness framework (Ben Allal et al., ) sampling 100 solutions with temperature $0.6$ and $p=0.95$.
The reported numbers on HumanEval and MBPP are obtained after tuning the $\beta$ values for each method on the $[0.1,0.2,0.4,0.6,0.8,1.0,5.0,10.0]$ set. The best $\beta$ is obtained based on the performance of each model on pass@10 with 10 samples on HumanEval.

## Appendix C Complementary Results

In this appendix, we provide results complementary to our experiments in Section 3.

### C.1 Sparsity and Token-level KL Divergence

We report the sparsity levels in the reward and divergence masks, for increasing values of $\beta$, over the rejected responses during training.
Figure 7 presents the case for sentiment control, whereas
Figure 8, for code generation.

Figure: Figure 7: Sparsity levels in the reward mask ($m_{u}$, left) and the token-level KL divergence mask ($m_{d}$, middle), as well as token-level KL divergence of rejected responses during training (over IMDB), for increasing values of $\beta$.
Refer to caption: x5.png

Figure: Figure 8: Sparsity levels in the reward mask ($m_{u}$, left) and the SeqKL divergence mask ($m_{d}$, middle), as well as SeqKL divergence of rejected responses during training (over MBPP), for increasing values of $\beta$.
Refer to caption: x6.png

### C.2 Qualitative Analysis

Figure 9 presents complementary results to Figure 4(b),
showcasing heatmaps for rejected response examples.

Figure: (a) Rejected response rewards.
Refer to caption: extracted/5910009/tdpov2_rejected_rewards.png

## Appendix D Ablation Studies

In this section, we present ablation studies that investigate the contribution of design choices in mask architectures.
All experiments were done by performing SFT and PO training on Pythia-410M using the DPO-mix-7k dataset curated by Argilla.(^12^1212[https://huggingface.co/datasets/argilla/dpo-mix-7k](https://huggingface.co/datasets/argilla/dpo-mix-7k))
This dataset consists of 7k instances mixed from Capybara(^13^1313[https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized)) a synthetic multi-turn dialogue dataset;
Intel ORCA(^14^1414[https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs)), a single-turn dataset based on FLAN, with prompts aiming at helpful, truthful, and verbalized calibration;
and the binarized, filtered version of UltraFeedback.(^15^1515[https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned))
Training was done for three epochs with learning rate of $5e-7$ and effective batch size of $128$ for all models.
Unless otherwise stated, all SparsePO systems were trained using the common mask setup.

### D.1 Mask Architecture

We experiment with the number of model layers used for mask calculation, as well as the number of feedforward layers in the mask architecture itself.
Table 3 showcases the performance of our design choices over benchmarks in the OpenLLM learderboard v2.

**Table 3: OpenLLM leaderboard v2 performance of mask architectural choices, for Pythia 410M-based models trained over DPO-mix-7k.**
| Lay.per Mask | #FFm | BBH | MATH | GPQA | MuSR | MLMU | IFEval | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  | pro | Instr. | Prom. |  |
| All Layers | 1 | 4.60 | 0.91 | 1.68 | 12.47 | 1.57 | 21.70 | 11.28 | 7.74 |
| Last Layer | 1 | 4.34 | 0.68 | 2.01 | 11.74 | 1.41 | 19.42 | 9.61 | 7.03 |
| Last Layer | 2 | 4.60 | 0.98 | 1.68 | 11.57 | 1.24 | 19.30 | 9.61 | 7.00 |

### D.2 Hyper-Parameter Tuning

Next, we investigate the effect of weight decay regularization applied over the mask, with results shown in Table 4.

**Table 4: OpenLLM leaderboard v2 performance for several levels of weight decay regularization over the mask.**
| Wgt. | BBH | MATH | GPQA | MuSR | MLMU | IFEval | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Decay |  |  |  |  | pro | Instr. | Prom. |  |
| 0 | 4.44 | 0.83 | 1.57 | 13.39 | 1.48 | 22.42 | 11.09 | 7.89 |
| 0.001 | 4.41 | 0.38 | 1.45 | 11.47 | 1.61 | 21.82 | 11.09 | 7.46 |
| 0.01 | 4.56 | 0.38 | 1.12 | 14.00 | 1.36 | 23.02 | 12.57 | 8.14 |
| 0.1 | 4.83 | 0.68 | 1.34 | 12.03 | 1.66 | 21.82 | 10.91 | 7.61 |
| 1.0 | 4.65 | 0.68 | 1.45 | 12.70 | 1.64 | 21.70 | 10.72 | 7.65 |

### D.3 Binary and Random Masks

Finally, we experiment with variations of SparsePO in which the learned mask is replaced by a uniformly-sampled random vector with values between $[0,1]$, and a learned binary mask with a $sign$ activation function, i.e. the mask is set to $1$ for all positive values and 0 0, otherwise.
Table 5 presents the results over the OpenLLM leaderboard.

**Table 5: OpenLLM leaderboard v2 performance for binary and random masks.**
| Mask | BBH | MATH | GPQA | MuSR | MLMU | IFEval | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | pro | Instr. | Prom. |  |
| SparsePO$[m_{u}=m_{d}]$ | 4.56 | 0.38 | 1.12 | 14.00 | 1.36 | 23.02 | 12.57 | 8.14 |
| SparsePO$[\text{Binary}]$ | 4.55 | 1.13 | 1.68 | 13.03 | 1.46 | 18.71 | 8.50 | 7.01 |
| Random | 4.84 | 0.68 | 1.34 | 14.49 | 1.33 | 20.26 | 9.61 | 7.51 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Although PPO is less used, many recent papers show that it is not inferior to DPO, so is it justified to exclude this from the evaluation?

Comments:
1. There are a few different terminology regarding SparsePO, including method, strategy, objective, and framework. This can be confusing to the reader.
soundness: 3
strengths: 1. The paper is clear and well-written.
2. The method seems novel, has a clear and well-established motivation, and is mathematically rigorous.
3. The paper performs experiments on a varied set of tasks.
4. The sentiment control experiments show good trade-offs between KL divergence and reward. The "Sparsity and Token-level KL divergence" experiment is insightful.
5. Nice improvements on IFEVAL and BBH with H&H training.
summary: The paper proposes SparsePO, a new approach for Preference Optimization (PO) with a token-level focus. Specifically, SparsePO uses sparse token masks to assign different weights to specific tokens, allowing flexible optimization. SparsePO uses dynamic or model-derived masking strategies.
Quantitative and qualitative experiments on a few varied tasks show that SparsePO works well and provides improvements in some of the cases.
weaknesses: 1. The  TL;DR dataset can be unfaithful and a small set of 120 prompts can hinder results further. Hence I tend to suspect the results. This is a more experimental design problem than a method problem. For faithfulness, AFAIK there are better methods like Q^2, True, GPM, and more.
2. Although the H&H shows some nice results, the size of the model combined with the difficulty of the benchmarks (OpenLLM-2 is designed to be much harder than 1) limit the ability to properly assess the method capabilities. Again this is more on the experimental design side. Running on the version may be better. Also, considering stronger models in the 1B range can also help (See Phi, SmolLM, Qwen2; see this blog reporting small LLMs results on OLLM-V.1 https://huggingface.co/blog/smollm).
3. The results on text-to-code are mixed or even negative raising questions regarding the versatility of the method.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Have you attempted to train the sparse mask using the Gumbel-Softmax function?
2. When the KL constraint is very small, a larger reward value obtained by the model usually implies a higher probability of reward hacking. Can you provide some case studies to illustrate the impact of SparsePO on preventing reward hacking?
soundness: 3
strengths: 1. The paper presents a technically sound approach. The motivation for proposing the objective function of SparsePO lies in the classic problem of token contribution allocation in reinforcement learning. The transformation is well-motivated and follows a logical progression.
 2. The use of masks to control the contribution of each token is a valid approach. The two proposed mask computation strategies, MAPO and SPARSEPO, are clearly described and seem feasible. The technical details provided in the methodology section, such as the equations for calculating the masks and the optimal policy, are sufficient to understand the implementation.
 3. In the experiments, the evaluation metrics used are appropriate for the tasks considered (sentiment control, dialogue, summarization, and text-to-code generation). The analysis of the trade-offs between reward and KL divergence, as well as the sparsity of the masks, provides a comprehensive understanding of the behavior of the proposed method.
summary: Previous studies have shown that specific tokens play a key role in learning desired behaviors during pre-training and preference optimization, especially in domains where preference depends on certain aspects or subsequences. Consequently, the authors introduce SparsePO, a method for sparse token-level preference optimization. This approach aims to learn sparse masks over token-level rewards and KL divergences during training. SparsePO offers flexibility, does not rely on external models, and can be combined with different masking methods. The authors also analyze the sparsity of induced masks and their relation to KL divergence, and demonstrate quantitative and qualitative improvements when applying SparsePO in various domains with preference indicators.
weaknesses: 1. Learned sparse masks do not necessarily match human preferences: In the learnable sparse mask, the author only illustrated in the paper how to adjust parameters to ensure the learned mask is sparse. However, it cannot be guaranteed that the crucial tokens are learned correctly. For example, in Figure 9(a), SparsePO-common rewards assigns almost equal rewards to all tokens.
 2. Inconsistent performance across metrics: Table 2 shows that SparsePO gains over pass@100 but has a slight decay in the remaining metrics. This indicates that while it may improve one aspect of code generation performance, it may not be uniformly beneficial across all evaluation criteria.
3. Difficulty in identifying important tokens across domains: In the code domain, especially for code execution, it is challenging to identify which particular tokens are more responsible for a program executing correctly. This is indicated by low mask sparsity levels, suggesting that the method may not be as effective in precisely weighting tokens for code generation as it could be for other domains.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the weakness part.
soundness: 3
strengths: 1. SparsePO introduces dynamic token weighting, enhancing model adaptability and generation diversity across different preference criteria.
2. The method is evaluated across multiple datasets.
summary: The paper introduces Sparse Preference Optimization (SparsePO), a novel approach for controlling large language models' preference alignment through sparse token masks. The authors note that current preference optimization methods (like DPO) treat all token weights equally, whereas human preferences often depend on specific words or phrases. SparsePO introduces flexible weight masks, enabling models to automatically learn weights for KL divergence and rewards during training, thereby improving adaptation to human preferences.
weaknesses: 1. The core motivation - that human preferences depend on specific words rather than equally on all tokens - lacks empirical and theoretical validation.
2.  The introduction of m(y_t) in Equation 3 does not guarantee optimization equivalence with previous work (Zeng et al., 2024).
3.  The learnable sparse mask implementation using a single feed-forward network requires more theoretical justification. Additionally, the method's sensitivity to model architecture and data distribution needs further examination, particularly for implicit alignment tokens.
4.  Section 3.3's evaluation focuses on reasoning tasks while omitting crucial assessments of helpfulness and harmlessness metrics, diminishing the significance of minimal performance degradation in reasoning.
5.  Section 3.4's summarization task evaluation would benefit from GPT-4-based win rate metrics. The improvements are marginal, with some metrics underperforming existing baselines.
6.  Section 3.5's experiments suffer from insufficient training data and show inferior performance compared to standard DPO, potentially contradicting the paper's claims.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: > Line 242: “TDPOv1 exhibits moderate KL divergence, which translates into higher reward than DPO and comparable to SimPO.” 
1. It looks to me that DPO attains substantially higher reward at KL <10 in Figure 2 than all other PO methods except MaPO. Could you explain?

2.  Could you explain what “TDPOv1 exhibits moderate KL divergence, which translates into higher reward than DPO ...” mean? Do you mean systems trained with TDPOv1 generally have KL < 20 at the end of TDPOv1 training? 

3. In stating “moderate KL translate into higher reward”, Are you suggesting a causal relationship between KL and reward? I understand KL and reward are two measures of system characteristics which could be correlated, but without causal relation.

> Line 302: “increasingly higher values of β induce higher levels of sparsity on the divergence mask (md), restricting the amount of tokens allowed to diverge in a sequence, which translates to lower token-level KL divergence throughout training.”

4. From Equation (3): high sparsity (i.e., more zero maskings) effectively drops out the KL term, allowing the policy to optimize the advantage function only. It seems like this increases the amount of tokens allowed to diverage in a sequence rather than restricting it. Could you help me understand how mask sparsity interferes with the policy objective? 

> Line 307: “we find that low values of β induce scenarios where reward sparsity is high and divergence sparsity is low, meaning that the loss is dominated by the masked divergence term, δ(x,y1,y2).”

5. Do you mean that the mask is restricting KL divergence at small beta? It seems like the mask is working against beta’s control of regularization. 
6. I am not sure what are the theoretical and emprical advantages of the proposed SPARSEPO methods. Could you provide a clear summary?
soundness: 2
strengths: * They select a broad set of experiments for investigating PO methods.
* The proposed token-level masking method is a natural extension of previous work on token-leval DPO.
summary: * This paper introduces token-level masking to the token-level DPO objective proposed by Zeng, et. a (2024)l. The motivation is that not all tokens should contribute to KL divergence and Reward computation equally. 
* They conduct experiments on standard tasks for preference optimization: Sentiment Control, Helpfulness & Harmlessness Control, TL;DR summarization, and Text-to-Code generation. 
* They analyze Sparisity level (i.e,. the amount of zero maskings) of the learned masks during training and discuss its implication.
* Their experimental results suggest that DPO remains a strong baseline. The proposed methods are marginally better, if at all than DPO, in terms of final performance or KL-Performance frontier,
weaknesses: * **Lack of precision in discussing results**. The discussion heavily relies on imprecise statements such as “TDPOv1 exhibits moderate KL divergence”. It is not clear how an PO algorithm could exhibit a certain level of KL divergence. This results in serious issues in clarity as to what the paper tries to argue with its experimental results (See Questions Below).
* **Proposed methods show marginal or no performance gain compared to DPO**. There is little evidence for adopting the proposed methods in terms of performance. For example, in sentiment control, DPO shows the best KL-Reward frontier at low KL (<10). Experiments on Text-to-Code (Table 2) essentially shows DPO remains the most robust PO algorithm. It is difficult to justify the complicated modeling techniques introduced by token-level masking based on their experimental results. 
* **Inadequate experimental setups**. While the paper includes a good range of tasks, some experimental setups are inadequate for investigating PO. For example, Table 1 on Helpfulness & Harmlessness show all PO methods underperforming SFT policy by substantial margin in terms of average scores on Open LLM Leaderboard 2. It is inadequate to argue which PO algorithm is best on the ground that it induces less degradation than SFT compared to other methods. In addition, Helpfulness & Harmfulness and Text-to-Code are investigated with models with <2B parameters. The baseline performance of these small models is poor on these tasks.
* **Advantages from introducing token-level masking are unclear**.  It is not clear from the paper what theoretical/empirical advantages there are for the proposed token-leval masks.

Meta-review: The paper proposes SparsePO, a preference optimization (PO) algorithm that aims to automatically learn to weight the KL divergence and reward corresponding to each token during PO training, motivated by the finding that human preference usually depends more on specific tokens rather than equally across all tokens. While the reviewers find the method to be interesting, they also raised several issues such as the insufficiently justified token mask designs and the marginal performance advantage over baselines.

Decision: Reject