SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks

Authors: Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou Ammar, Jun Wang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: preference optimization, alignment with human preferences

Abstract:
Preference Optimization (PO) has proven an effective step for aligning language models to human-desired behaviors. Current variants, following the offline Direct Preference Optimization objective, have focused on a strict setting where all tokens are contributing signals of KL divergence and rewards to the loss function. However, human preference is not affected by each word in a sequence equally but is often dependent on specific words or phrases, e.g. existence of toxic terms leads to non-preferred responses. Based on this observation, we argue that not all tokens should be weighted equally during PO and propose a flexible objective termed SparsePO, that aims to automatically learn to weight the KL divergence and reward corresponding to each token during PO training. We propose two different variants of weight-masks that can either be derived from the reference model itself or learned on the fly. Notably, our method induces sparsity in the learned masks, allowing the model to learn how to best weight reward and KL divergence contributions at the token level, learning an optimal level of mask sparsity. Extensive experiments on multiple domains, including sentiment control, dialogue, text summarization and text-to-code generation, illustrate that our approach assigns meaningful weights to tokens according to the target task, generates more responses with the desired preference and improves reasoning tasks by up to 2 percentage points  compared to other token- and sentence-level PO methods.

Decision: reject
Reviewer scores: 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730203571677

thread_type:
official_review

<thread_object>
creator: reviewer_hu2n

<content>
summary:
* This paper introduces token-level masking to the token-level DPO objective proposed by Zeng, et. a (2024)l. The motivation is that not all tokens should contribute to KL divergence and Reward computation equally. 
* They conduct experiments on standard tasks for preference optimization: Sentiment Control, Helpfulness & Harmlessness Control, TL;DR summarization, and Text-to-Code generation. 
* They analyze Sparisity level (i.e,. the amount of zero maskings) of the learned masks during training and discuss its implication.
* Their experimental results suggest that DPO remains a strong baseline. The proposed methods are marginally better, if at all than DPO, in terms of final performance or KL-Performance frontier,

strengths:
* They select a broad set of experiments for investigating PO methods.
* The proposed token-level masking method is a natural extension of previous work on token-leval DPO.

weaknesses:
* **Lack of precision in discussing results**. The discussion heavily relies on imprecise statements such as “TDPOv1 exhibits moderate KL divergence”. It is not clear how an PO algorithm could exhibit a certain level of KL divergence. This results in serious issues in clarity as to what the paper tries to argue with its experimental results (See Questions Below).
* **Proposed methods show marginal or no performance gain compared to DPO**. There is little evidence for adopting the proposed methods in terms of performance. For example, in sentiment control, DPO shows the best KL-Reward frontier at low KL (<10). Experiments on Text-to-Code (Table 2) essentially shows DPO remains the most robust PO algorithm. It is difficult to justify the complicated modeling techniques introduced by token-level masking based on their experimental results. 
* **Inadequate experimental setups**. While the paper includes a good range of tasks, some experimental setups are inadequate for investigating PO. For example, Table 1 on Helpfulness & Harmlessness show all PO methods underperforming SFT policy by substantial margin in terms of average scores on Open LLM Leaderboard 2. It is inadequate to argue which PO algorithm is best on the ground that it induces less degradation than SFT compared to other methods. In addition, Helpfulness & Harmfulness and Text-to-Code are investigated with models with <2B parameters. The baseline performance of these small models is poor on these tasks.
* **Advantages from introducing token-level masking are unclear**.  It is not clear from the paper what theoretical/empirical advantages there are for the proposed token-leval masks.

questions:
> Line 242: “TDPOv1 exhibits moderate KL divergence, which translates into higher reward than DPO and comparable to SimPO.” 
1. It looks to me that DPO attains substantially higher reward at KL <10 in Figure 2 than all other PO methods except MaPO. Could you explain?

2.  Could you explain what “TDPOv1 exhibits moderate KL divergence, which translates into higher reward than DPO ...” mean? Do you mean systems trained with TDPOv1 generally have KL < 20 at the end of TDPOv1 training? 

3. In stating “moderate KL translate into higher reward”, Are you suggesting a causal relationship between KL and reward? I understand KL and reward are two measures of system characteristics which could be correlated, but without causal relation.

> Line 302: “increasingly higher values of β induce higher levels of sparsity on the divergence mask (md), restricting the amount of tokens allowed to diverge in a sequence, which translates to lower token-level KL divergence throughout training.”

4. From Equation (3): high sparsity (i.e., more zero maskings) effectively drops out the KL term, allowing the policy to optimize the advantage function only. It seems like this increases the amount of tokens allowed to diverage in a sequence rather than restricting it. Could you help me understand how mask sparsity interferes with the policy objective? 

> Line 307: “we find that low values of β induce scenarios where reward sparsity is high and divergence sparsity is low, meaning that the loss is dominated by the masked divergence term, δ(x,y1,y2).”

5. Do you mean that the mask is restricting KL divergence at small beta? It seems like the mask is working against beta’s control of regularization. 
6. I am not sure what are the theoretical and emprical advantages of the proposed SPARSEPO methods. Could you provide a clear summary?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730322876811

thread_type:
official_review

<thread_object>
creator: reviewer_6tmp

<content>
summary:
The paper introduces Sparse Preference Optimization (SparsePO), a novel approach for controlling large language models' preference alignment through sparse token masks. The authors note that current preference optimization methods (like DPO) treat all token weights equally, whereas human preferences often depend on specific words or phrases. SparsePO introduces flexible weight masks, enabling models to automatically learn weights for KL divergence and rewards during training, thereby improving adaptation to human preferences.

strengths:
1. SparsePO introduces dynamic token weighting, enhancing model adaptability and generation diversity across different preference criteria.
2. The method is evaluated across multiple datasets.

weaknesses:
1. The core motivation - that human preferences depend on specific words rather than equally on all tokens - lacks empirical and theoretical validation.
2.  The introduction of m(y_t) in Equation 3 does not guarantee optimization equivalence with previous work (Zeng et al., 2024).
3.  The learnable sparse mask implementation using a single feed-forward network requires more theoretical justification. Additionally, the method's sensitivity to model architecture and data distribution needs further examination, particularly for implicit alignment tokens.
4.  Section 3.3's evaluation focuses on reasoning tasks while omitting crucial assessments of helpfulness and harmlessness metrics, diminishing the significance of minimal performance degradation in reasoning.
5.  Section 3.4's summarization task evaluation would benefit from GPT-4-based win rate metrics. The improvements are marginal, with some metrics underperforming existing baselines.
6.  Section 3.5's experiments suffer from insufficient training data and show inferior performance compared to standard DPO, potentially contradicting the paper's claims.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454939687

thread_type:
official_review

<thread_object>
creator: reviewer_5ghz

<content>
summary:
Previous studies have shown that specific tokens play a key role in learning desired behaviors during pre-training and preference optimization, especially in domains where preference depends on certain aspects or subsequences. Consequently, the authors introduce SparsePO, a method for sparse token-level preference optimization. This approach aims to learn sparse masks over token-level rewards and KL divergences during training. SparsePO offers flexibility, does not rely on external models, and can be combined with different masking methods. The authors also analyze the sparsity of induced masks and their relation to KL divergence, and demonstrate quantitative and qualitative improvements when applying SparsePO in various domains with preference indicators.

strengths:
1. The paper presents a technically sound approach. The motivation for proposing the objective function of SparsePO lies in the classic problem of token contribution allocation in reinforcement learning. The transformation is well-motivated and follows a logical progression.
 2. The use of masks to control the contribution of each token is a valid approach. The two proposed mask computation strategies, MAPO and SPARSEPO, are clearly described and seem feasible. The technical details provided in the methodology section, such as the equations for calculating the masks and the optimal policy, are sufficient to understand the implementation.
 3. In the experiments, the evaluation metrics used are appropriate for the tasks considered (sentiment control, dialogue, summarization, and text-to-code generation). The analysis of the trade-offs between reward and KL divergence, as well as the sparsity of the masks, provides a comprehensive understanding of the behavior of the proposed method.

weaknesses:
1. Learned sparse masks do not necessarily match human preferences: In the learnable sparse mask, the author only illustrated in the paper how to adjust parameters to ensure the learned mask is sparse. However, it cannot be guaranteed that the crucial tokens are learned correctly. For example, in Figure 9(a), SparsePO-common rewards assigns almost equal rewards to all tokens.
 2. Inconsistent performance across metrics: Table 2 shows that SparsePO gains over pass@100 but has a slight decay in the remaining metrics. This indicates that while it may improve one aspect of code generation performance, it may not be uniformly beneficial across all evaluation criteria.
3. Difficulty in identifying important tokens across domains: In the code domain, especially for code execution, it is challenging to identify which particular tokens are more responsible for a program executing correctly. This is indicated by low mask sparsity levels, suggesting that the method may not be as effective in precisely weighting tokens for code generation as it could be for other domains.

questions:
1. Have you attempted to train the sparse mask using the Gumbel-Softmax function?
2. When the KL constraint is very small, a larger reward value obtained by the model usually implies a higher probability of reward hacking. Can you provide some case studies to illustrate the impact of SparsePO on preventing reward hacking?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723009044

thread_type:
official_review

<thread_object>
creator: reviewer_jkpm

<content>
summary:
The paper proposes SparsePO, a new approach for Preference Optimization (PO) with a token-level focus. Specifically, SparsePO uses sparse token masks to assign different weights to specific tokens, allowing flexible optimization. SparsePO uses dynamic or model-derived masking strategies.
Quantitative and qualitative experiments on a few varied tasks show that SparsePO works well and provides improvements in some of the cases.

strengths:
1. The paper is clear and well-written.
2. The method seems novel, has a clear and well-established motivation, and is mathematically rigorous.
3. The paper performs experiments on a varied set of tasks.
4. The sentiment control experiments show good trade-offs between KL divergence and reward. The "Sparsity and Token-level KL divergence" experiment is insightful.
5. Nice improvements on IFEVAL and BBH with H&H training.

weaknesses:
1. The  TL;DR dataset can be unfaithful and a small set of 120 prompts can hinder results further. Hence I tend to suspect the results. This is a more experimental design problem than a method problem. For faithfulness, AFAIK there are better methods like Q^2, True, GPM, and more.
2. Although the H&H shows some nice results, the size of the model combined with the difficulty of the benchmarks (OpenLLM-2 is designed to be much harder than 1) limit the ability to properly assess the method capabilities. Again this is more on the experimental design side. Running on the version may be better. Also, considering stronger models in the 1B range can also help (See Phi, SmolLM, Qwen2; see this blog reporting small LLMs results on OLLM-V.1 https://huggingface.co/blog/smollm).
3. The results on text-to-code are mixed or even negative raising questions regarding the versatility of the method.

questions:
1. Although PPO is less used, many recent papers show that it is not inferior to DPO, so is it justified to exclude this from the evaluation?

Comments:
1. There are a few different terminology regarding SparsePO, including method, strategy, objective, and framework. This can be confusing to the reader.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731966184034

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
We would like to thank all reviewers for their time in reviewing the manuscript and providing their insightful
comments. Here, we will be addressing comments that are common among reviewers and provide some
additional results to support the paper’s thesis. We are also currently performing additional experiments to
address reviewers concerns; we will be adding these results as they become available.
To further support SparsePO’s efficacy compared to related work and concerns raised by multiple reviewers, we
provide win-rates for the HH and TL;DR benchmarks using GPT-4 (gpt-4-turbo) as the judge on 100 randomly
chosen prompts of the test set. In particular, we compare the gold chosen response with a response generated
by our model after PO training and ask GPT-4 to select the best response for various temperatures. The
prompts used follow Rafailov et al. [2023] and Amini et al. [2024] (we will also include those in an updated
version of the paper by the end of the response period). As observed from the tables below, in HH, SparsePO-
common obtains the best win-rates across the board with SparsePO-indp being the close second. For TL;DR,
systems are close with MaPO showing the highest rates.
Given that multiple reviewers raised that the benefits of our proposed method were unclear, we summarize
them below. Please note that these benefits take into account the additional results we provide. 

With respect to the paper’s contributions:
- Flexibility in increasing or suppressing the contribution of each token during preference optimization.
- Our method produces a non-trivial level of sparsity that confirms, alongside the reported performance
gains, that not all tokens are required for effective preference optimization.
- Using token-level masks, models are injected with information that allow them to recognize which tokens
are crucial for the target proxy, potentially enabling external signals as well. Here, we simply focused on
either using the reference model activations or learning a mask via learned parameters.
- Our method is able to produce higher rewards at larger KL divergence values enabling models to generate
more diverse responses.
- We are able to achieve improved performance in terms of gpt-4 evaluations on HH and TL;DR bench-
marks.

Furthermore, our analysis has shown:
- Performance-wise SparsePO/MaPO have achieved: +0.91 pp on OpenLLM Leaderboard-v2, overall with
+1.32 and +2.03 specifically on instruction following benchmarks.
- +10% winrates in Anthropic-HH single-turn dialogue (using GPT-4-turbo as a judge across different
temperatures).
- +3% winrates in TL;DR summarization (using GPT-4-turbo as a judge across different temperatures).
With respect to the results on Text-to-code generation, we aim to re-frame it as negative case for our method,
where a sparse approach cannot work due to the nature of the task, hence, wee provide the following intuitions.

The thesis behind SparsePO is that human preference is often reliant on specific words or phrases in sentences,
which is well observed and supported in Natural Language. However, code sequences are heavily structured
and every ‘word’ is intricately reliant on all other ‘words’ in the sequence, i.e. there is little information that
may be considered redundant. As such, a weighing scheme will effectively ignore parts of the sequence that
can be crucial. Additionally, in natural language PO is mostly used to reduce toxicity or improve style, but
in code it is strictly to improve functional correctness, and ignoring any ‘word’ in a code sequence will most
certainly lead to a functionally incorrect solution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732790387726

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Latest Version Updates

comment:
We would like to thank all the reviewers for their insightful feedback that helped us significantly improve the paper.
Based on their original comments and responses, we have updated the paper to include additional experiments as well as clarifications to strengthen our work. We have highlighted the corresponding places with a different color for their convenience. We also briefly include the changes here:
- Win-rates for HH and TL;DR with GPT-4 as a judge, in Figures 5 and 7.
- Analysis in Appendix C.1 with respect to the reward and response-level KL trade-off [Reviewer hU2n]
- Updated results over TL;DR after filtering low faithfulness instances in Figure 6 and Appendix C.5 [Reviewer JkpM]
- Results on the Open LLM Leaderboard v1 in Appendix C.4 [Reviewer JkpM]
- More accurate description of the relationship between KL divergence and expected reward in the Pareto frontier in lines 232-246 [Reviewer hU2n]
- Further intuitions about text-to-code results in Section 3.5.
- Discussion about how the value of $\beta$ works in tandem with mask values in lines 260-263 and Appendix C.2 [Reviewer hU2n].
- Further analysis regarding the correlation between mask distribution and token-level KL divergence in Appendix C.3 [Reviewer hU2n]
- Our code is now available (check Appendix B.1)
- We further evaluated Pythia-1.4B on the HumanRankEval test set, results provided in Appendix C.7.
- Additional qualitative results for HH, MATH and code in Appendix C.6, Figure 22.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes SparsePO, a preference optimization (PO) algorithm that aims to automatically learn to weight the KL divergence and reward corresponding to each token during PO training, motivated by the finding that human preference usually depends more on specific tokens rather than equally across all tokens. While the reviewers find the method to be interesting, they also raised several issues such as the insufficiently justified token mask designs and the marginal performance advantage over baselines.