LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Authors: Bowen Jin
Jinsung Yoon
Zhen Qin
Ziqi Wang
Wei Xiong
Yu Meng
Jiawei Han
Sercan O Arik
Venue: ICML 2025 poster (2025)
Keywords: information retrieval
LLM alignment
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) have revolutionized artificial intelligence with capabilities in reasoning, coding, and communication, driving innovation across industries. Their true potential depends on effective alignment to ensure correct, trustworthy and ethical behavior, addressing challenges like misinformation, hallucinations, bias and misuse. While existing Reinforcement Learning (RL)-based alignment methods are notoriously complex, direct optimization approaches offer a simpler alternative.
In this work, we introduce a novel direct optimization approach for LLM alignment by drawing on established Information Retrieval (IR) principles. We present a systematic framework that bridges LLM alignment and IR methodologies, mapping LLM generation and reward models to IR's retriever-reranker paradigm. Building on this foundation, we propose LLM Alignment as Retriever Preference Optimization (LarPO), a new alignment method that enhances overall alignment quality. Extensive experiments validate LarPO's effectiveness with 38.9 % and 13.7 % averaged improvement on AlpacaEval2 and MixEval-Hard respectively. Our work opens new avenues for advancing LLM alignment by integrating IR foundations, offering a promising direction for future research.

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Bowen Jin 12 Jinsung Yoon?
Zhen Qin
Ziqi Wang'
Wei Xiong
Yu Meng 4
Jiawei Han
Sercan Ö. Arik? 2
Abstract
tial of LLMs while ensuring their ethical, safe, and high- quality performance hinges effective alignment (Wang et al., 2023). However, existing reinforcement learning- based LLM alignment methods (e.g., PPO (Ouyang et al., 2022)) involve multi-stage training and are challenging to optimize. To this end, direct LLM preference optimization methods (e.g., DPO (Rafailov et al., 2024)) are proposed to simplify the alignment process.
Large Language Models (LLMs) have revolution- ized artificial intelligence with capabilities in rea- soning, coding, and communication, driving in- novation across industries. Their true potential depends on effective alignment to ensure correct, trustworthy and ethical behavior, addressing chal- lenges like misinformation, hallucinations, bias and misuse. While existing Reinforcement Learn- ing (RL)-based alignment methods are notori- ously complex, direct optimization approaches offer a simpler alternative. In this work, we in- troduce a novel direct optimization approach for LLM alignment by drawing on established In- formation Retrieval (IR) principles. We present systematic framework that bridges I.M align- ment and IR methodologies, mapping LLM gener- ation and reward models to IR's retriever-reranker paradigm. Building on this foundation, we pro- pose LLM Alignment as Retriever Preference Optimization (LARPO), a new alignment method that enhances overall alignment quality. Exten- sive experiments validate LARPO's effectiveness with 38.9 % and 13.7 % averaged improvement on Alpa2aEval2 and MixEval-Hard respectively. Our work opens new avenues for advancing LLM alignment by integrating IR foundations, offering a promising direction for future research.
In this work, further enhance direct LLM preference op- timization, focusing on bringing Information Retrieval (IR) perspectives (Tay et al., 2022). Striking parallels exist be- tween IR methodologies and LLM aligmment techniques (Lin et al., 2022). For example, IR's retriever-rerant framework, which uses retriever for broad semantic match- ing to generate candidate grained refinement, offers a compelling analogy to the Best- of-N approach in LLM alignment (Dong et al., 2023; Sessa et al., 2024). In this analogy, the LLM acts as the retriever, while the reward model serves as the reranker. Further- more, the common use of dual-encoder architectures in both LLM generation and IR retrievers, coupled with the reliance on cross-encoder architectures in reward models and IR rerankers, further underscores this synergy. Leveraging es- tablished IR techniques offers the potential to develop novel, easily implementable LLM alignment methods grounded in IR principles, leading to improved alignment quality.
and reranker for fine-
Despite the promising connections between LLM alignment and IR. systematic exploration of this synergy remains lacking. Specifically, three key gaps exist: (1) a clear map- ping between LLM alignment mechanisms and core IR prin- ciples has not been established; (2) empirical evaluations of LLMs through an IR lens are scarce; and (3) proven IR techniques like retriever optimization, hard negative mining, and candidate list construction are underutilized for LLM alignment. This paper directly addresses these gaps by sys- tematically bridging LLM alignment and IR methodologies. Our contributions are fourfold:
1. Introduction
Large Language Models (LLMs) (Achiam et al., 2023; Team al., 2024a) have demonstrated remarkable capacities in wide range of fields including conversational modeling (Zhao et al., 2023a), reasoning (Wei et al., 2022) and code generation (Jiang et al., 2024). Unlocking the full poten-
'University of Illinois at Urbana-Champaign GGoogee Cloud Al Research Google DeepMind "University Virginia. Corre- spondence to: Bowen Jin <bowenj4 @ illi.ois.eeu>..
We introduce comprehensive framework that connects LLM alignment techniques with the established IR princi- ples, providing perspective on LLM alignment.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We demonstrate the significance of three key IR prin-

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
ciples retriever optimization objectives, hard negative mining, and candidate list construction for improving LLM alignment.
2.2. LLMs as retrievers. Reward models as rerankers
During inference, an LLM generates a response y given input prompt by modeling the probability distribu- tion PLLM(y(æ)) Assuming fixed maximum sequence length L. and a vocabulary space V (Li et al., 2024), the set of all possible responses can be defined as Y = { y 1 (2)
Building these insights, we propose novel align- ment method, LLM Alignment Retriever Preference Optimization (LARPO), which demonstrably enhances alignment quality, with 38.9 % and 13.7 % relative aver- aged improvement on AlpacaEval2 and MixEval- Hardd
E V] S V
We can conceptualize this process through an IR lens (Tay et al., 2022). The prompt x can be viewed as analogous to query q, the set of all possible responses Y can be treated the corpus C, and the generated response y can be considered as the retrieved passage d. Thus, given a prompt x, the L.M effectively acts as a retriever, searching for the most probable responses (LLM(Z) from response space Y:
We conduct further empirical studies to evaluate LLM performance using IR metrics, analyzing the impact of various post-training techniques.
In summary, this work establishes a crucial link between IR and LLM alignment, offering both novel insights and practical methods for advancing the field.
DLLM(#) (y E Y max pL)M(d
top-K
where (L)M(y|z) is analogous to Pretriesa in IR.
2. An Information Retrieval Perspective on LLMs
This analogy is further supported by the LLMs' architec- ture. As illustrated in Figure 1, the generative modeling with LLMs can be interpreted as the matching process of bi-encoder model. The prompt is encoded into a vector representation by LLM, while response tokens are repre- sented as token embedding vectors. For each token position decoding, prompt embedding (obtained often from the hid- den state of the last layer of the LLM) and vocabulary token embeddings are compared with dot product, determine the likelihood of selected token for the response.
2.1. Primer on information retrieval
Information retrieval systems (Zhu et al., 2023) typically employ two-stage process involving retrievers (Zhao et al., 2024) and rerankers (Lin et al., 2022). The retriever, often implemented as bi-encoder (Figure 1), efficiently identifies large set of (K) potentially relevant passages, denoted as Dretrieal, from a corpora C' given a query This is achieved using a coarse- granied similarity function, Preuieva = Ency y (4) ())-Enaa(),, where Enc, and Enc, represent the query and passage encoders respectively:
Furthermore, reward models (2,( (Lambert et al., 2024), which take both the prompt and response as in- put, function similarly to cross-encoders (i.e., rerankers Frank (q, d) (Zhuang et al., 2023)) in IR. To enhance L.M performance, various inference-time strategies have been developed, including Best-of-N sampling (Stiennon et al., 2020) and majority voting (Wang et al., 2022). These can be interpreted as different configurations of retrievers and rerankers, as summarized in Appendix Table 5.
(1etiooal(g) {dd C' max Pretrievaa(dlg)))) top-K
(L)
However, due to the scale of the corpus, retrievers might not accurately capture fine-graing ane-grained query-passage similarity with the simple dot production interaction function. There- fore, rerankers, typically implemented with cross-encoder (Figure 1), are employed refine the ranking of the re- trieved passages Drenicval- The reranker produces a smaller set (k) of top-ranked passages, Drank using fine- -naiined similarity function, Frank(9.0) d) w- Enc(q, d), where w is a learnable linear layer. Here, reranker adopts cross- encoder with both query/passage as inputs and encoded together while retriever adopts dual encoder for separate query/passage encoding.
2.3. LLM tuning as retriever optimization
Supervised fine-tuning as direct retriever optimization. Retriever training, aiming for accurate retrieval, often em- ploys contrastive learning with the InfoNCE loss (Oord et al., 2018) to maximize P(dgota g)) of retrieving the ground truth passage dgold given a query q. This can be expressed as:
maxlog P(dgotalg) = max log Enca(dgoua) Enc,(g)
Drank( (g) (dl (ret max top-k
maan Trank (g, d)3-
(nn Enc,
The resulting ranked passages are ordered such that Drank = (d1, d2..d..... .....,d d+} where Trank d1) >
In the context of LLM alignment, supervised fine-tuning (SFT) aims to quickly adapt the model to a target task us- ing prompt-response pairs (r, Mgota). SFT maximizes the
... Trank (g. d2) 2 ..... 2 Prank( dz).
2

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
architecture
Bl-encoder
Cross-encoder
tasks
QUIDY ancoder
passigll encadeeeenccerr encoder
jont
score (1)
Information retrieval
query
passage (d)
query
passage (d)
(a) retriever: Pretriezat (dlq)
(b) reranker: Frank (q.
vocab embodding
ancoddrr
score,
Generative language
modeling
prompt (x)
response [y)
prompt (x)
respónse (y)
(c) LLM: PILM (y/x))
(d) reward model: Tym (x.
Figure 1. Architecture connection between retriever/LLLM (bi-encoder) and reranker/reward model croos-ennodder.. Bi-encoder models process cach query/prompt and passage/response separately and often calculate their alignment score via dot product operator, while cross-encoder models take both query/prompt and passagefresponse as input and score them directly. Bi-encoder models can more efficient (i.e., large-scale text matching) but the interaction between the two information unit is only captured by dot production operation where their effectiveness can constrained. Cross-encoder models can more effective (i.e.. deeper interaction calculation with transformer architecture (Vaswani, 2017)) but less efficient. Although L.M involves auto-regressive token matching, which is different from retriever, some insights from IR can borrowed enhance L.M alignment as shown in the following sections.
conditional probability P( (r) as:
(Schulman et al., 2017) and iterative DPO (Guo et al., 2024) exemplify this approach. Here, the LLM (considered acting as the retriever) generates responses that are then scored by the reward model (considered acting as the reranker). These scores are used to create new training data, effectively performing distillation from the reward model into the LLM:
Mastel
max log P(Ugolalz) = max log
P ((Ugota())
lygasl
Emb( (ygola (1)) LLL((2))
max >
log
feward-model S data gf-) fLLM(-)- Thus, preference opti- mization can be viewed form of to-rereeett distillation, analogous to the process used in traditional IR.
Σ E um)( (2,)
where v(i) is the i-th token of y, [a,, Mlgold [1: i 1)] represent the concatenation of the prompt æ and the preceding tokens of Ygold, L.M(() produces contextualized representation, and Emb(-) is the token embedding function. We assume vocab embeddings and LLM hidden states share the same dimension, as in most LLMs.
We conduct empirical studies to understand SFT and prefer- ence optimization from IR perspective in Appendix B and have further discussion in Appendices C and D.
2.4. Empirical insights into LLMs as IR models
Consequently, the SFT objective can be interpreted as composite of multiple retrieval optimization objectives. In this analogy. LLM(-) acts as the query encoder and Emb(-) serves as the passage (or, in this case, token) encoder.
Evaluating LLMs as retrievers. A common metric for evaluating retrievers is Recall N, which assesses whether the top-N retrieved passages include any relevant passages for given query. In the context of LLMs, this translates to evaluating whether the top-N generated responses contain suitable response to the prompt, analogous to Pass @ N (Chen et al., 2021).
Preference optimization reraiver-retriever distilla- tion. In retriever training, optimizing solely based on query/grounddtruth document pairs can be suboptimal, par- ticularly when using in-batch negatives for efficiency. Per- formance can enhanced by distilling knowledge from a more powerful reranker retriever (Qu al., 2020: Zeng al., 2022). This distillation process be represented
To draw the empirical connection between LLM and re- trievers, we conduct an experiment on the GSM8K dataset (Cobbe et al., 2021) using Mathstral-7b-it (Mistral A1, 2025) and an experiment the NQ dataset (Kwiatkowski et al., 2019) using e5 retriever. Figure 2 illustrates that increasing N can contribute to improved performance for both retriever and LLM. Detailed analysis can be found in Appendix E.
as freran((-) (-) S. data 2() fretrioval-). where new data, gener- ated by the reranker frerank(-) based rule is used to optimize the retriever Srerrevat (.) with an objective (-)).
Greedy decoding, equivalent to N = 1, is a prevalent LLM inference strategy. However, as shown in Figure 2(b), allow- ing multiple attempts (N 1) can substantially improve the chance of producing correct answer, suggesting that per-
Similarly, in LLM alignment, preference alignment phase often follows supervised fine-tuning (SFT) to further en- hance the model using an external reward model to ab- sorb preferential supervision effectively. Methods like PPO
3

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Table LLM alignment objectives of LARPO. In the table, 7(v x:) | log xyer(y(a)" All the proofs be found in Appendix F.
Ta(yla)
Method
Assumption of r(z,y)
Objective
DPO
Pr( > w)) a(((a, Bw) n(2,
Lpur
-E
log o
log relyulal log malpcla) Fear (bulF)
Feer(wl)))
LARPO (Contrastive)
Pr(yw in Mi (1)
Me 2 Di m/m))) softmax(r((,,
Leaa
log
exp( (xp(((u))))) exp (op(()))))) - cop (1t0/"
LARPO (lambdaRank)
Pr(y) ... 2 Vm)
Te a -(x,
Ltamb
-F
Σ Leteepeo log (7(8) 12) 7(36 1
(T(2n()))))
LARPO (LisIMLE)
Th = softmaxx" (r(x,, Pr(y) 2n Vm)
Limle
E
log
exp (atule))))) +21 exp (1(0)1)))
performance (Xiong et al., 2024; Xu et al., 2024b; Guo et al., 2024) (Figure 3(b).. Drawing inspirations from retriever optimization, we re-examine iterative LLM preference opti- mization, focusing on three key aspects: (1) the optimization objective; (2) the use of hard negatives; and (3) the candi- date list construction. Based on these aspects, we propose a LLM alignment with an IR perspective, LARPO.
0.95
1.0
0.9
0.85
00.80
PassiN
0.8
0.75
0.70
0.7
0.65
0.6
0.60
0.5
100 150 200 250 300
20
80 100
* retrieved passages (N)
generated responses (N)
(a) Retriever
(b) LLM
3.1. Retriever optimization objective
Figure 2. Analogy between evaluating retriever with Recall N LLM with Pass@N. As the number (N) of retrieved pas- sages/generated responses increases, the retriever and LLM have similar increasing trend. This highlights the importance of infer- ence time scaling (e.g., Best-of-N) for LLM similar to retriever- reranker scaling in IR. Retriever: e5; LLM: Mathstral- -b-ii..
Typical objectives for retriever optimization include pair- wise, contrastive and listwise objectives (Zhao et al., 2024). In this section, we discuss preference optimization vari- ants (Wang et al., 2023) corresponding to different retriever optimization objectives. The optimization objective for pref- erence optimization is given as:
formance under N may underestimate the model's full potential. This highlights the importance of inference-time scaling techniques like Best-of-N (Stiennon et al., 2020) in LLM similar retriever-reranker scaling (Zhuang al., 2023) in IR. More results and analyses can be found in Appendix E.
max Ex, (e y)] BBLLB (- ULLM
As discussed in Rafailov et al. (2024), the equation above has the optimal solution as:
r(z, y) = Glog TLLM(U Teet (ular) + Blog
(4)
3. Iterative LLM alignment as retriever optimization
where 2 == Ey, Teet ( (xx (rz y²)) ) is the normaliza- tion constant and r(.) is the reward model which can also be seen as a reranker. According to different assumption for r(a, y) from IR, we can obtain different training objectives shown in Table 1, with proofs in Appendix
Optimization (t.g., pairwisc, constrestive, Usswise)
Optimization
Queries/ Corpora
prompts !
Labeling
retriever
fhand negationsl
retrieve
responses
labeling
LLM
goneraae
Pairwise ranking. Under the pairwise -rrdlyy-Terry) assumption Fr( 2 vi) ((((x,).) - r(z, vt),, the policy objective becomes DPO (Rafailov et al., 2024) Caai
reranker processing
reward model
selection responses
passages
(candidate lizt)
(e) Iterative retriever opti mization
(b) Iterative LLM alignment
Contrastive ranking. Another widely used objective for ranking contrastive learning (Oord et al., 2018):
Figure 3. The connection between iterative LLM alignment (Xiong et al., 2024) and iterative retriever optimization (Xiong et al., 2020)
Pr(yo 2 Vi Var 2 1) (m)) softmax (r(x, vw))
Iterative learning is common technique in retriever opti- mization (Xiong et al., 2020), where results from the newly- trained model are used to generate new training data, as illustrated in Figure 3(a). Similarly, for LLM alignment, iterative preference optimization has been shown to enhance
exp(r(x, yw))
(5)
exp(r(x, yw)) + > exp(r(x,))
It handles multiple negatives in a single step, allowing the model learn more robust representations for retrieval

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
and ranking. It is widely used for dense retriever training (Karpukhin al., 2020). Under this ranking assumption, the policy objective becomes Loo shown in Table 1.
Algorithm LARPO: LLM alignment iterative retriever preference optimization.
Require: Number of iterations T, number of new data per annotation phase M, number of generated responses for each prompt k, temperature for each iteration {t+) prompt dataset Dx {a, J policy LLM Ia, reward model r, learning rate 7 ranking- bbaed objective function Lrank.
Lank In addition to pairwise and contrastive learning, list-wise ranking is widely adopted to sufficiently utilize the comprehensive information in candidate list. In- spired by LambdaRank (Burges, 2010; Zeng et al., 2022):
Ensure: Aligned LLM mBr
Pr(y- 2 ... ym)
I 1<iajam
o(r(z, 81) r(z, 82)),
1: for s == 0 T do
Update behavior LLM: TT + TT0
3: Preference dataset D, {}
the policy optimization objective becomes Llamb (Table 1).
4: for i :- 1 to M do
5: Sample prompt ~ Dx
Another list-wise ranking assumption is the ListMLE assumption (Xia et al., 2008), which provides theoretical grounding and global optimization perspective:
ListMLE.
6: // candidate list construction
7:
Sample V1..... yk ~ -z(-1z)), hard negatives
9: Rank {vs} with r: Yv = (u,"'), where (r(s)))
Pr(y1 2 2 ym))
I softmax,""((e vi))
r(vb (r) )),a <b
10:
D, - D. u 1(x, Y²))
exp(r(e, vi)
11: end for
exp(r(x, y:)) + -5 exp(r(x,
12:
candidate list construction D Merge
13:
In this case, the objective becomes Limtt shown in Table 1.
14:
while D Ø do
15: Sample a batch (x, Y2) from D
3.2. Hard negatives
16:
Update D - Dx, Yx)) // retriever optimization objective
17:
Hard negatives are crucial for effective retriever training
18:
0₂ < 0g y Vo (I, Yz, No: Tg))
(Zhan et al., 2021; Qu al., 2020), learning to distin-
19:
end while
guish harder negatives potentially lead to more powerful
20: 0,+1 0s
retrievers (Xiong et al., 2020). In LLM alignment, negatives correspond to unpreferred responses (yi) for given prompt (x). In iterative on-policy training, various types of nega- tives can be identified, ordered by increasing difficulty: (1) Easiest:
21: end for
size and candidate selection as being particularly important. Similarly, in iterative preference optimization, construction of the candidate response list Y [V1, ... VmmJ is critical. We identify two key factors influencing the quality of Y: inclusiveness and memorization.
random, unrelated response to x; (2) Easy: A response a related but different prompt (x'); (3) Hard: An incorrect response generated with high temperature; (4) Hardest: An incorrect response to generated with low temperature.
Note that, assuming a well-initialized policy LLM, as in- dicated by Figure 2(b) (N = 1), low temperatures tend to produce harder negatives, yielding the above ranking, To be specific, lower temperatures yield more similar generated responses, increasing overlap between positive and negative samples. This effectively makes the negatives harder. Ac- cording to Zhan et al. (2021), hardest negatives could be most important to LLM alignment.
(1) Inclusiveness (Qu et al., 2020) refers the size of the response list Y. A larger Y potentially encompasses more information.
(2) Memorization (Zeng et al., 2022) refers whether previ- ously generated responses Y' are included in the current list Y to preserve past results.
Given their importance in IR (Qu et al., 2020; Zeng et al., 2022), the impact of these factors on LLM alignment, how- ever, remains largely under-explored.
3.3. Candidate list
In iterative retriever optimization, construction of the can- didate list [d]...., d which is used by the reranker to generate data for the next iteration, is crucial. Prior re- search (Zeng et al., 2022) has identified factors such as list
4. The Proposed Solution: LARPO
Motivated by iterative retriever optimization pipeline as
5

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Table 2. Evaluations on AlpacaEval 2 and MixEval. I.C. WR and WR denote length-controlled win and win respectively. Offline baseline performances on AlpacaEval 2 are from Meng et al. (2024b). We use LLM-blender (Jiang et al., 2023b) as the reward model for a fair comparison with the baselines and also report the result with a stronger reward model FsfairX (Dong et al., 2024)
Model
Mistral-Base (7B)
Mistral-Instruct (7B)
Alpaca Eval 2
MixEval
MixEval-Hard
Alpaca Eval
MixEval
MixEval-Hard
LC WR
WR
Score
Score
LC WR
WR
Score
Score
SFT
8.4
6.2
0.602
0.279
17.1
14.7
0.707
0.361
Reward model:
(Jiang al., 2023b)
RRHF
11.6
10.2
0.600
0.312
25.3
24.8
0.700
0,380
SLiC-HF
10.9
8.9
0.679
0.334
24.1
24.6
0.700
0.381
DPO
15.1
12.5
0.686
0.341
26.8
24.9
0.702
0.355
IPO
11.8
9.4
0.673
0.326
20.3
20.3
0.695
0.376
CPO
9.8
8.9
0,632
0.307
23.8
28.8
0.699
0,405
KTO
13.1
9.1
0.704
0.351
24.5
23.6
0.692
0.358
RDPO
17.4
12.8
0.693
0.355
27.3
24.5
0.695
0.364
SimPO
21.5
20.8
0.672
0.347
32.1
34.8
0.702
0,363
Iterative DPO
18.9
16.7
0.660
0.341
20.4
24.8
0.719
0.389
LARPO (Contrastive)
31.6
30.8
0.703
0.409
32.7
38.6
0.718
0.418
LARPO (1 Rank
34.9
37.2
0.695
0,452
32.9
38.9
0.720
0.417
LARPO (I.ss
31.1
32.1
0.669
0.390
29.7
36.2
0.709
0.397
Reward model: FsfairX (Dong al., 2024)
LARPO (Contrastive)
41.5
42.9
0,718
0.417
43.0
53.8
0.718
0.425
LARPO (LambdaRank)
35.8
34.1
0.717
0.431
41.9
48.1
0.740
0.440
LARPO (ListMLE)
36.6
37.8
0.730
0.423
39.6
48.1
0.717
0.397
shown in Figure 3(a) and the three key points in IR, we introduce LARPO, a novel approach to LLM alignment for- mulated as iterative retriever preference optimization. The algorithmic details are provided in Algorithm 1. Specif- ically, our experimental setup explores the following key aspects: (1) Optimization objective: We evaluate three distinct loss functions as the ranking objective (Crank): foom. flamb, and Limie. (2) Hard negatives: For given prompt, hard negative samples are constructed by selecting less pre- ferred responses generated with an appropriate temperature through parameter search. More details of the temperature are available in Appendix H.1. (3) Candidate list: In each iteration, we generate multiple (10) candidate responses considering inclusiveness. In terms of memorization, the candidate pool for subsequent iterations includes all previ- ously generated responses.
are from Meng al. (2024b). Further details regarding these baselines and our experimental setup are provided in Appendix G. Both baselines and LARPO are trained on Ultrafeedback dataset (Cui et al., 2024) for fair comparison.
Datasets. We conduct evaluation on two widely used benchmarks AlpacaEval2 (Dubois et al., 2024) and Mix- Eval (Ni et al., 2024). These benchmarks are designed to assess the conversational capabilities of models across diverse range of queries. AlpacaEval2 comprises 805 questions sourced from five datasets, while MixEval in- cludes 4000 general and 1000 hard questions. Evaluation follows the established protocols for each benchmark. For AlpacaEval 2, we report both the raw win rate (WR) and the Jength-controlle win rate (LC). These benchmarks collec- tively provide a comprehensive assessment the models" cssruction-folloviii and problem-solving capabilities.
5. Main Results
Baselines. We evaluate the performance of LARPO against a range of established preference optimization meth- ods, encompassing both offline and online approaches. Our offline comparison set includes RRHF (Yuan et al., 2023), SLiC-HF (Zhao et al., 2023b), DPO (Guo et al., 2024), IPO (Azar et al., 2024), CPO (Xu et al., 2024a), KTO (Ethayarajh et al., 2024), RDPO (Park et al., 2024) and SimPO (Meng ct al., 2024b). For online methods, we compare with iter- ative DPO (Xiong et al., 2024). The baseline checkpoints
Results. The baseline performances on AlpacaEval 2 are directly from Meng et al. (2024b), while the performances on MixEval is evaluated by ourselves with the opensourced checkpoints. We adopt the same LLM-Blender (Jiang et al., 2023b) reward model for fair comparison with the base- lines and also explore stronger reward model: FsfairX (Dong et al., 2024). The results, presented in Table 2, show that LARPO consistently outperforms the competitive baseline methods on both datasets, with 38.9 % and 13.7 % aver- aged relative improvements, on AlpacaEval2 and MixEval-
6

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Hard respectively, with the same reward model as the base- lines. With stronger reward model, can further im- prove LARPO by 25.8 % on the challenging AlpacaEval2 dataset. Additional details regarding our experimental setup are available in Appendix H.1.
designated as the positive example. The impact of differ- ent hard negative variants is investigated, as described in Section 3.2, with the results presented in Figure 4(a). Addi- tionally, the influence of temperature on negative hardness with Lambdarank objective are examined using experiments on the AlpacaEval 2 dataset, with results shown in Figure 4(b). Detailed settings are in Appendix H.5 and H.6.
6. Analyses
This section provides empirical analyses of the three factors identified in Section 3.
Observation. Figure 4(a) illustrates that the effectiveness of the final LLM is directly correlated with the hardness of the negatives used during training. Harder negatives consistently lead to a more performant LLM. Figure 4(b) further demonstrates that, within a specific range, lower temperatures generate harder negatives, resulting effective final trained LLM. However, much lower tempera- ture could lead to less diverse responses and finally lead to LLM alignment performance drop.
6.1. Retriever optimization objective
Experimental setting. Iterative preference optimization is performed on LLMs using the different learning objec- tives outlined in Section 3.1. Alignment experiments are conducted using the Gemma2-2b-it (Team et al., 2024b) and Mistral-7bbit (Jiang et al., 2023a) models, trained on the Ultrafeedback dataset (Cui et al., 2024). Following the methodology of (Dong et al., 2024), we conduct three iter- ations of training and report the performance of the final checkpoint in Table 3. Model evaluations are performed AlpacaEval2 (Dubois et al., 2024) and MixEval (Ni et al., 2024). Detailed settings can be found in Appendix H.2.
more
6.3. Candidate List
Experimental setting. To investigate the impact of inclu- siveness and memorization on LLM aligmment, experiments are conducted using Gemma2-2b-is employing the same training settings as in our objective study. For the inclusive- ness study, the performance of the trained LLM is evaluated using varying numbers of candidates in the list. For the memorization study, three approaches are compared: (i) using only the current iteration's 's responses, (ii) using re- sponses from the current and previous iteration, and (iii) using responses from the current and all previous iterations.
Table Preference optimization objective study on AlpacaEval2 and MixEval. SFT corresponds to the initial chat model.
AlpacaFxvll 2
MixEval
MixFval-Hand
Method
LC Winrate
Winrate
Score
Score
Gemma2 2b.it
SFT
36.39
38.26
0.6545
0,2980
pairwise
41.39
54.60
0.6740
0.3375
Detailed settings can be found in Appendix H.7 and H.3.
contrastive
43.41
56.83
0.6745
0,3315
ListMLE
49.77
62.05
0.6715
0,3560
LambdaRank
43.76
60.56
0.6750
0,3560
Mistral- 7b-it
SFT
21.14
14.22
0.7070
0.3610
Table 4. Candidate list study with Cpar on Gemma2-2b-i Previous iteration responses enhance performance.
pairwise
36.43
41.86
0.7175
0.4105
contrastive
38.44
42.61
0,7260
0.4340
Alpaca Eval
ListMLB
38.02
43.03
0.7360
0.4200
LambhdRank
40.29
46.21
0.7370
0.4400
Method
I.C. Winrate
Winrate
SFT
47.03
48.38
Observation. Table 3 presents the results, from which we make the following observations: (I) Contrastive opti- mization generally outperforms pairwise optimization (e.g.. DPO), likely due to its ability to incorporate more negative examples during each learning step. (2) Listwise optimiza- tion methods, including ListMLE and LanbdaRank, gen- erally demonstrate superior performance compared to both pairwise and contrastive approaches. This is attributed to their utilization of a more comprehensive set of preference information within the candidate list.
Aligmment (w. current)
55.06
66.56
Alignment (w. current + prev)
55.62
70.92
Alignment (w. current all prev)
56.02
72.50
Observation. Figure 4(c) illustrates the significant impact of candidate list size on LLM aligmment performance. As the candidate list size increases, performance improves, al- beit with a diminishing rate of return. This is intuitive, given that a bigger candidate list size can contribute to more hard negatives and potentially benefit the model learning (Qu et al., 2020). Table 4 demonstrates that incorporating re- sponses from previous iterations can enhance performance. This is potentially because introducing previous responses can make the candidate list more comprehensive and lead to better preference signal capturing. More explanations are in Appendix H.3.
6.2. Hard negatives
Experimental setting. The Mathstral-7b-it model is trained on the GSM8k training set and evaluated its per- formance on the GSM8k test set. Iterative DPO is employed as the RLHF method, with the gold or correct response
7

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
0.84
[4] Hardest
64
0.82
[3) Hard
(2) Easy
60
62
(1) Easiest
60
0.80
58
Rabe
LC winrate
188
0.78
56
166
0.76
54
34
52-
LC winrate
0.74
52
50
Winrate
0.80
1.00 1.05 1.10 1.15 *Temperature
1.20
10
Iteration
Respanses
(a) Hard negative study
(b) Temperature & hard negatives
(c) Candidate list length study
Figure 4. Hard negative and candidate list study. (a) Hard negative study with Cpair on GSM8K with Mathstral-7b-i model. We explore
four negative settings: (1) random response not related to the given prompt; (2) a response to a related prompt: (3) an incorrect response to the given prompt with high temperature: (4) an incorrect response to the given prompt with suitable temperature. Hardness: (2)>()))))))))), The harder the negatives are, the stronger the trained LLM is. (b) Training temperature study with Guir on Mistral-7b-it and Alpaca Eval 2. Within a specific range (> 1), lower temperature leads to harder negative and benefit the trained LLM. However, much lower temperature could lead to less diverse responses and finally lead to LLM alignment performance drop. (c) Candidate list size study with Ceon on Mistral-7b-it. As the candidate list size increases, alignment performance improves.
7. Related works
data, which is hard to satisfy in practice.
LLM alignment. Pretrained LLMs demonstrate remark- able capabilities across broad spectrum of tasks (Brown et al., 2020). Their performance at downstream tasks, such as conversational modeling, is significantly enhanced through alignment with human preferences (Ouyang et al., 2022; Bai et al., 2022). RLHF (Christiano et al., 2017) has emerged as a foundational framework for this alignment, typically involving learning a reward function via prefer- model (Bradley & Terry, 1952), and tuning the LLM using reinforcement learning (RL) to optimize this reward. Despite its success, RLHF's practical implementation is notoriously complex, requiring multiple LLMs, careful hyperparameter tuning, and navigating challenging optimization landscapes.
Language models for information retrieval. Language models (LMs) have become integral to modern IR sys- tems (Zhu et al., 2023), particularly after the advent of pretrained models like BERT (Devlin, 2019). A typical IR pipeline employs retrievers and rerankers, often based on dual-encoder and cross-encoder architectures, respec- tively (Humeau, 2019). Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) pioneered the concept of dense retrieval, laying the groundwork for subsequent research. Building on DPR, studies have emphasized the importance of hard negatives in training (Zhan et al., 2021; Qu et al., 2020) and the benefits of online retriever optimization (Xiong et al., 2020).
ence model, often using the
Recent research has focused on simplifying this process. A line of works studies the direct alignment algorithms (Zhao et al., 2023b; Rafailov et al., 2024; Azar et al., 2024), which directly optimize the LLM in supervised manner without first constructing a separate reward model. In particular, the representative DPO (Rafailov et al., 2024) attracts signifi- cant attention both academia and industry. After these, SimPO (Meng et al., 2024b) simplifies DPO by using length regularization in place of reference model.
In the realm of reranking, Nogueira & Cho (2019) were among the first leverage pretrained language models for improved passage ranking. This was followed by MonoT5 (Nogueira et al., 2020), which scaled rerankers using large encoder-decoder transformer architectures, and RankT5 (Zhuang et al., 2023), which introduced pairwise and list- wise ranking objectives. Recent work has also highlighted the importance of candidate list preprocessing before rerank- ing (Meng et al., 2024a).
Although LLMs are adopted for IR (Tay et al., 2022), there is a lack of study to improve direct LLM alignment with IR principles. This paper fills this gap by establishing a system- atic link between LLM alignment and IR methodologies, and introducing novel iterative L.MM alignment approach that leverages insights from retriever optimization to ad- vance the state of the art. The most related work is LiPO (Liu et al., 2024), which applies ceaming-to-rank objectives. However, LiPO relies on off-the-shelf listwise preference
Despite the pervasive of LMs in IR, the interplay be- tween LLM alignment and IR paradigms remains largely unexplored. This work aims to bridge this gap, establishing a strong connection between LLM alignment and IR, and leveraging insights from both fields to advance our under- standing of L.M alignment from an IR perspective.
8

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
8. Conclusions
Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint N.Xiv:2204.05862, 2022.
This paper has forged novel link between LLM alignment and IR, offering a systematic framework to enhance the LLM alignment performance. Expanding upon this basis, we introduced LARPO, a new direct preference optimiza- tion method that integrates the IR principles to significantly enhance aligmment quality. The effectiveness of LARPO is strongly supported by our comprchensive experiments across uidely-used benchmarks, demonstrating its potential as a significant advancement I.LM alignment. Further- more, our IR-focused analysis highlights the crucial role of retriever optimization objectives, hard negatives, and candidate list construction in achieving effective alignment.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-3444 1952.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Burges, C. J. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81, 2010.
Acknowledgements
This research was supported in part by Apple PhD Fellow- ship, in part by US DARPA INCAS Program No. HR001 21-C0165 and BRIES Program No. HR0011-24-35 1-00011-24330322 1-24-3-0325, in part by the Office of Naval Research contract number N000142412612, in part by NSF grant numbers IIS-19- 56151 and 2402873, in part by the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897 and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 21 2118329, in part by Cisco, and in part by the Center for Intelligent Information Retrieval. Any opinions, findings, and conclu- sions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of the sponsors or the U.S. Govern- ment.
Chen, M.. Tworek, J., Jun, H., Yuan, Q.. Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.0337 2021.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110 14168, 2021.
Cui, G., Yuan, L., Ding. N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
Impact Statement
Devlin, J. Bert: Pre-training of deep bidirectional transform- ers for language understanding. In NAACL, 2019.
This paper contributes to the advancement of machine leam- ing. While our work may have broader societal implications, we do not believe any specific impacts warrant explicit dis- cussion in this context.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06766 2023.
References
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf arXiv preprint 20t:::4000078633 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Dubois, Y., Galambosi. B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model aligmment as prospect theoretic optimization. arXiv preprint arXiv:2402.0 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint (rXii:2402.04792,, 2024.
Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., and de Rijke, M. Ranked list truncation for large language model-based re-ranking. In Proceedings of the 47th In- teratiionll ACM SIGIR Conference on Research and De- velopment in Information Retrieval, pp. 141-151, 2024a.
Humeau, S. Poly-encoders:
Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv: 1905.01 01650199,,,00 2019.
Meng. Y., Xia, M., and Chen, Simpo: Simple preference
optimization with a reference-free free reward. arXiv preprint arXiv:2405. 2024b.
Jiang, A. Q., Sablayrolles, A., Mensch, A.. Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel. G., Lample, G., Saulnier, I.., et al. Mistral 7b. arXiv preprint arXiv:2310.06880 2023a.
Mistral AL. Introducing mathstral, 2025, URL https: // mi mistral.at peii/neww ht/////w////aaa http////wwtatttaall/// /. Accessed: 2025- 01-16.
Jiang, D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.0 025661,,, 2023b.
Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neu- big, G., and You, Y. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures. arXiv preprint arXiv:2406. 06565, 2024.
Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. A survey on large language models for code generation. arXiv preprint arXiv:2406.0057 2024.
Nogueira, R. and Cho, K. Passage re-ranking with bert. arXiv preprint 1rXiv:1901.000
2019.
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-. Dense passage retrieval for open-domain question answering. arXiv preprint Skkv:22004049006, 2020.
Nogueira, R., Jiang, Z., and Lin, J. Document ranking with a pretrained sequencc-to--eeueecee model. arXiv preprint arXiv:2003.06713, 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint (00Yv:/807.0:3330 2018
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for ques- tion answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 1122773-277444 2022.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024.
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., and Dou, Z. From matching to generation: A sur- vey on generative information retrieval. arXiv preprint arXiv:2404. 14851, 2024.
Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W.X., Dong, D., Wu, H., and Wang, H. Rocketqa: An optimized train- ing approach to dense passage retrieval for open-domain question answering, arXiv preprint arXiv:2010.08191, 2020.
Lin, J., Nogueira, R., and Yates, A. Pretrained transformers for text ranking: Bert and beyond. Springer Nature, 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36. 2024.
Liu, T. Qin, Z., Wu, J., Shen, J., Khalman, M., Joshi, R., Zhao, Y., Saleh, M., Baumgartner, S., Liu, J., et al. Lipo: Listwise preference optimization through learning- to-rank. arXiv preprint arXiv:2402.01878 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1007:0::777.06444 2017.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreifee S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieil- lard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A.,
10

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Cideron, G., et al. Bond: Aligning Ilms with best-of-n distillation, arXiv preprint arXiv:2407.14622 2024
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., and Overwijk, A. Approximate nearest neigh- bor negative contrastive learning for dense text retrieval. arXiv preprint heti::0007000006 2020.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 2020.
Xiong, W., Dong, H., Ye, C.,, Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Con- ference on Machine Learning, 2024.
Tay, Y. Tran, V., Dehghani, M., Ni, J., Bahri, D., Mehta, H., Qin, Z., Hui, K., Zhao, Z., Gupta, J., et al. Trans- former memory as differentiable search index. Ad- vances in Neural Information Processing Systems, 35: 21831-21843, 2022.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.0844 2024a.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, I.., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint ar arXiv:2403.05530 2024a.
Xu, W. Li, J., Wang, W. Y., and Li, L. Bpo: Staying close to the behavior IIm creates better online Ilm alignment. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11125- 11 11139, 2024b.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXhv:2408.00114 2024b.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li. Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 00.9/.2222 12284, 2023.
Yuan, Z., Yuan, H., Tan, C., Wang. W., Huang, S., and Huang, Rrhf: Rank responses to align language mod- els with human feedback without tears. arXiv preprint arXiv:
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Huang, S., Rasul, K., Bartolome, A., M. Rush, A., and Wolf, T. The Alignment Handbook. URL https: /github.. com/hugging wwtaaaaeaaaaagiaaaaeeee http//iiinnntaaabbbooooo
2023.
Zeng, H., Zamani, H., and Vinay, V. Curriculum learning for dense retrieval distillation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1979-1983, 2022.
Vaswani, Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M.. and Ma. S. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, pp. 1503-1512, 2021.
Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey, arXiv preprint arXiv:2307. 120.229976.666.. 2023.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.182. 2023a.
Wei, J., Wang, X.. Schuurmans, D.,, Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thoughh prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Zhao, W. X., Liu, J., Ren, R., and Wen, J.-R. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1-60, 2024.
Xia, F., Liu, T.-Y., Wang, J., Zhang, W., and Li, H. List- wise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pp. 1192-1191 2008
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprins arXiv:2305. 10425, 2023b.

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Zhu, Y., Yuan, H., Wang. S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z.,, Dou, Z., and Wen, J.-R. Large language models for information retrieval: A survey, arXiv preprint arXiv:2308.07107, 2023.
Zhuang, H., Qin, Z., Jagerman, R., Hui, K., Ma, J., J., Ni, J., Wang, X., and Bendersky, M. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2308-
2313, 2023.
12

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
A. LLM inference strategy and IR pipelines
Table 5. Correspondence between LLM inference IR pipelines.
Method
Retriever
Reranker
Pipeline
Greedy decoding
LLM
Ø
Retriever-only
Best-of-N (Stiennon al., 2020)
LLM
Reward model
Retriever-reranker
Majority voting (Wang al., 2022)
LLM
Majority
Retriever-reranker
Iterative refinement (Madaan al., 2024)
LLM
Iterative retrieval query rewriting
B. How can SFT and preference optimization help the LLM from an IR perspective?
We assess how well LLMs perform two tasks: fine- fine-grained reranking (using greedy decoding accuracy) and coarse -rainerr retrieval (using Recall @ N). We focus how SFT and DPO, affect these abilities. Using the Mistral-7b model, evaluate on the GSM8K and MATH datasets with two approaches: SFT-only, and SFT followed by DPO (SFT - DPO).
In the SFT phase, the model is trained directly on correct answers. For DPO, we generate 20 responses per prompt and created preference pairs by randomly selecting one correct and one incorrect response. We use hyperparameter tuning and early stopping to find the best model checkpoints (see Appendix H.4 for details).
Table Retrieval (Recall@N) and reranking (greedy accuracy) metrics across dataset and training strategies, with Mistral-7b the LLM. 0.7 is used as the temperature. Recall @ can also be denoted as pass@N.
Metric
init model
SFT
SFT DPO
GSM8K
Greedy Acc
0.4663
0.7680
0.7991
Recall 20
0.8347
0,9462
0.9545
Recall@50
0.9090
0,9629
0,9727
Recall 100
0.9477
0.9735
0,9826
Greedy Acc
0.1004
0.2334
0,2502
Math
Recall @ 20
0.2600
0,5340
0,5416
Recall 50
0.3354
0.6190
0.6258
Recall 100
0.4036
0.6780
0.6846
The results are shown in Table 6. We observe that both SFT and DPO improve both retrieval and reranking, with SFT being more effective. Adding DPO after SFT further improves performance on both tasks. This is consistent with information retrieval principles that both direct retriever optimization and ceranker-rotrieval distillation can enhance the retriever performance, while the latter on top of the former can further improve the performance. Further discussions can be found in Appendices C and D.
C. Discussion on the connection and difference between SFT and direct retriever optimization
As discussed in Section 2.3, the direct retriever optimization goal with InfoNCE is shown as:
max log P(dgotalq) max log Enca(dgows) Ency
Σ Enca(d)) Ency(g)
while the SFT optimization goal is shown as:
Monal
Emb ygold (1)) LLM(2,)
max log P(ygotal (r) = max log I
((ygota(i)) 1x:===
(i)lz)) IT1x
Σ
log
(8)
Emb(r;) LLM(2:)
As result, the SFT objective can be seen as a summation of multiple retrieval optimization objectives, where LLM(() and word embedding Emb(-) are query encoder and passage encoder respectively.
13

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
However, for direct retriever optimization with InfoNCE, Ency(-) is usually large-scale pretrained language model which computationally expensive both time and memory. In this case, it is unrealistic to calculate the Enca(dy) for all dj C, when C' is large, because of the time constrain and GPU memory constrain. As result, a widely-adopted technique is to adopt "in-batch negatives" with "hard negatives" to estimate the log P(dgouag)) function:
max log P(dgolal4) qq) = max log Enca(dgou) Enca(g) max log Enca(d:) Enc,(g) + Σ Enca(dj) Enc,(g) 2/=1 Enca(d,) (nc,(())
Enca(dgola) Enc,(g)
where B is the in-batch negative set and H is the hard negative set. Note that B BHH < C. This objective is more efficient to optimize but is not the original optimization goal. As a result, the learned model after direct retriever optimization is not optimal. It is also found that the hard negatives H is the key to estimate the original optimization goal (Zhan et al., 2021). Thus, reranker-retriever distillation can further improve the retriever by introducing more hard negatives.
On the other hand, LLM optimization, as shown in Eq. (8), can be seen as a summation of multiple retrieval optimization function. In each retrieval step, the passage can be seen as a token and the corpus is the vocabulary space V. Given that the passage encoder Emb(-) (word embedding) here is cheap to compute and the vocabulary space V (<110kk is usually not as large as C' (>> IM) in IR, the objective in Eq. (8) can be directly optimized without any estimation. In this case, the LLM as a retriever is more sufficiently trained compared with the retriever training in IR.
D. Discussion on the connection and difference between preference optimization and reranker-retriever distillation
As discussed in Section 2.3, preference optimization with an online reward model froward-model - data 2 fLLM(-) can be seen as reranker to retriever distillation process fremank (-) _ data 2(2 fretreval (+), where the reward model is the reranker (i.e., cross-encoder) and the LLM the retriever (i.e., bi-encoder). However, there are two slight differences here:
The LLM after SFT is more sufficiently trained compared a retriever after direct optimization. As discussed in Appendix C, the SFT optimization function is not an estimated retriever optimization goal compared with the direct retrieval optimization. As result, the LLM after SFT is suffienctly trained. In this case, if the reward model (reranker) cannot provide information other than that already in the SFT set (e.g., using the SFT prompts), this step may not contribute to significant I.LM capability improvement.
The reward model may introduce auxiliary information than the reranker in IR. For reranker in IR, it captures same semantic with the retriever: semantic similarity between the query and the passage. However, in LLM post-training, the goal and data in SFT and preference optimization can be different. For example, the SFT phase could have query/response pairs which enable basic chat-based retrieval capability for the LLM. While the reward model may contain some style preference information or safety information which do not exist in SFT data. In this case, the preference optimization which is the reranker to retriever distillation step could also contribution to performance improvement.
E. Evaluate LLMs as retrievers
In addition to Mathstral- ib-it on GSM8K in Figure 2, we conduct extensive experiments to both Mistral-7b-it and Mathstral- 7b-it on GSM8K and MATH. The results are shown in Figure 5. We have similar findings as in Figure 2 that: (1) As N increases, Recall improves significantly, indicating that retrieving a larger number of documents increases the likelihood of including a correct one within the set. (2) For smaller values of (e.g., N = 1), lower temperatures yield higher Recall @ N. This is because lower temperatures reduce response randomness, favoring the selection of the most relevant result. (3) Conversely, for larger N (e.g., N 10), higher temperatures enhance Recall N. Increased temperature promotes greater response diversity, which, when combined with a larger retrieval set, improves the chances of capturing the correct answer within the results.
F. LARPO retriever optimization objective
We provide the proof for different variants of LARPO's objective functions.
14

---

LLM Alignment Retriever Optimization: An Information Retrieval Perspective
Recallint (PaSS
Recallew (PasS8
0.8
0.8
0.7
temperature-O
0.6
temperature-
temperature-0
temperature-02
0.5
temperature-0.5
temperature-03
0.6
temperature=0.77
temperature=00
temperature=1.
temperature=10
100
(a) Mistral-7b-itt on GSM8k
(b) Mistral-7b-ic on GSM8k
0.5
0.7
Recalig (PassioN)
Recallg (PasseN)
0.6
0.5
temperature-0. 1
0.3
temperature-0.. 0.1
temperature=0.2
temperature=02 0.2
0.4
temperature=0.5
0.2
temperature=
temperature=0.7
temperature-0.7
temperature-1.0
temperature-11. 1.0
20
60
100
60
100
N
N
(c) Mathstral-7h-it on MATH
(d) Mistral-7b-it MATH
Figure Evaluate the LLM as retriever with R (Pass the number (N) of retrieved responses increases, retrieval recall increases. The higher the temperature is, the broader spectrum the retrieved responses are, and thus the higher the recall is.
F.1 Contrastive ranking
Theorem F.1. Let be prompt and (yws (1) .... Vi (m)) be the responses for under the contrastive assumption (Eq. Then the objective function to learn the LLM zg:
exp(>(Uw xr)
Loon B
log log
exp (2(w x)) Σi exp (xp()>(/)) x))
(9)
7lv x) Blog TT( x)
where
Tre( (y/ x))
Proof. From (Rafailov et al., 2024), we know that
Tilm (5|z)
r(z, y) Bogg
Tree (y(r) + :logZ,
(10)
where 7 Ey Teall Tnrll'(a))xpp (r(x, s/)).
15

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Then,
Pr(yee 2 Vi yv 2 11 m/) softmax(r(z,, yw:))
exp(r(x,, yw))
(xp(r(x,)w)) m exp(r(x, (0))
1+ exp(r ( r(2, vwol
1
(11)
1+ exp(r(1( ( 2x) + BlogZ (Bo x) - -Blog2)
1+ E exp(1(1)) (8) x)) - 7( xx)
cxp(>(Vw x))
exp(1(Uw) x)) Σx-1 exp (>(()) (1) x))
We can learn Te by maximizing the logarithm-likeii likelihood:
maxlog Pr(y z v/ ...... yw 2 3i m/) min log Pr 2 3/ ...... yu zim
L,
(12)
exp x)) exp(>(11a x)) + Ei exp(>(1)) (0) |
:. Leon
E log
(13)
7( x) = Bog To( x)
where
(14)
Tre x))
F.2. LambdaRank ranking
Theorem F22. Let be a prompt and (y1...., Vm) be the responses for æ under the LambdaRank assumption (Eq,(6)). Then the objective function to learn the LLM Ng:
o (7(88) 2)) - 700 | x1) ]
Ljamb -
Σ leiijem
log
(15)
Proof.
Fr(y1 ... vm)
II beiejem
a(r(z, vi) ((x,8))))
Il l<i<j<m
a(1(x, 14) + Blog% 7(x,, wj) Blog2)
(16)
I I<iej<m
o(1(3 x) 7125 x)).
We can learn ma by maximizing the logarihmn-lkkelhoodd:
max log Pr(y Vi (1) ..... Un 2 Vi (m)
min log Pr(y 2 11 (1)
,..., vi mi) L,
(17)
log (7(91 (x) 1z)--in >70; 11)
:. Liamb -E
(18)
7(1 x)) = Blog Toly x))
where
(19)
Tred (y x)
F.3. ListMLE ranking
Theorem F33 Let x be a prompt and (yi...., ym) the responses for x under the ListMLE assumption (Eq.(7)). Then the objective function learn the L.M no:
16

---

LLM Alignment Retriever Optimization: An Information Retrieval Perspective
exp(> x)) log exp(r(Vi x)) + EE= exp(>(35 x))
Limk
E8
Σ
(20)
Proof. From Eq.(1
11
Prr(y) 2 2 ymm
Pr(y: 2 Ditl, M.. 2 ymm
(21)
exp(>(yi xx)
exp(>(y x)) + 2== exp(>(1) x))
We can learn my by maximizing the logarithm-likelinne
laxlog Pr
Vi
(1)
.....
yw 2 Vi (mm) min - log Pr(y V1 (1)
...., yv vi (m)) L.
(22)
exp(1 |/ xx))
.' Limle
E
log
(23)
exp(1((1 |x)) + 2j=it1 exp (rus xx)
To x) Teet (y x)
where
7yy x) = 3og
(24)
G. Baselines
We conduct detailed illustrations on the baselines compared with LARPO in Section below.
RRHF (Yuan et al., 2023) scores responses via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss.
SLiC-HF (Zhao et al., 2023b) proposes sequence likelihood calibration method which can learn from human preference data.
DPO (Guo et al., 2024) simplifies the PPO (Ouyang et al., 2022) algorithms into offline direct optimization objective with the pairwise Bradley-Terry assumption.
IPO (Azar al., 2024) theoretically grounds pairwise assumption in DPO into a pointwise reward.
CPO (Xu et al., 2024a) adds reward objective with sequence likelihood along with the SFT objective.
KTO (Ethayarajh et al., 2024) adopts the Kahneman Tversky model and proposes method which directly maximizes the utility of generation instead of the likelihood of the preferences.
RDPO (Park et al., 2024) modifies DPO by including an additional regularization term to disentangle the influence of length.
SimPO (Meng et al., 2024b) further simplifies the DPO objective by using the average log probability of sequence the implicit reward and adding target reward margin to the Bradley-Terry objective.
Iterative DPO (Xiong al., 2024) identifies the challenge of offline preference optimization and proposes iterative learning framework.
H. Experiment settings
H.1. Table 2
We conduct evaluation on two widely used benchmark: AlpacaEval2 (Dubois et al., 2024) and MixEval (Ni et al., 2024). We consider two base models: Mistral-7b-bart and Mistral-7b-it. For Mistral-7b-bart we first conduct supervised finetuning following Meng et al. (2024b) before the preference optimization.
17

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Table Preference optimization objective study on AlpacaEval2 and MixEval. For AlpacaEval2, report the result with both opensource
LLM evaluator 1tt.llammmam00___mm and GPT4 evaluator wtwp::awww.......t........ SFT corresponds to the
initial chat model.
AlpacaEval 2 (opensource LLM)
AlpacaEvvll (GPT-4)
MixEval
MixEval-Hard
Method
LC Winrate
Winrate
LCC Winrate
Winrate
Score
Score
SFT
47.03
48.38
36.39
38.26
0.6545
0.2980
Gemma2-2b-i
pairwise
55.06
66.56
41.39
54.60
0.6740
0.3375
contrastive
60.44
72.35
43.41
56.83
0.6745
0.3315
ListMLE
63.05
76.09
49.77
62.05
0.6715
0.3560
LambdaRank
58.73
74.09
43.76
60.56
0.6750
0.3560
SFT
27.04
17.41
21.14
14.22
0.7070
0.3610
Mistral- 7b-it
pairwise
49.75
55.07
36.43
41.86
0.7175
0,4105
contrastive
52.03
60.15
38.44
42.61
0.7260
0,4340
ListMLE
48.84
56.73
38.02
43.03
0.7360
0.4200
LambdaRank
51.98
59.73
40.29
46.21
0.7370
0.4400
The performance scores for offline preference optimization baselines are from SimPO (Meng al., 2024b). To have fair comparison with these baselines, we adopt the same off-the-shelf reward model (Jiang et al., 2023b) as in SimPO for the iterative DPO baseline and LARPO.
For the iterative DPO baseline, we generate 2 responses for each prompt, score them with the off-the-shelf reward model and construct the preference pair data to tune the model.
For LARPO (contrastive Ceen), we generate 10 responses each iteration and score them with the reward model. The top-1 ranked response and the bottom-3 ranked responses adopted as the chose response and rejected responses respectively. Generation temperature is selected as 1 and 0.8 for Mistral-7b-base and Mistral-7b-it respectively (we search it among 0.8, 0.9, 1.0, 1.1, 1.2).
For LARPO PO (LambdaRank Llamb), we generate 10 responses each iteration and score them with the reward model. The top-2 ranked response and the bottom-2 ranked responses are adopted as the chose response and rejected responses respectively, Generation temperature is selected as 1 and 0.8 for 0.9, 1.0, 1.1, 1.2).
and Mistral-7b-i respectively (we search it among 0.8,
For LARPO (ListMLE Limie), we generate 10 responses each iteration and score them with the reward model. The top-2 ranked response and the bottom-2 ranked responses are adopted as the chose response and rejected responses respectively. Generation temperature is selected as I and 0.8 for Mistral-7b-be -b-base and Mistral-7b-it respectively (we search it among 0.8, 0.9, 1.0, 1.1, 1.2).
LARPO can achieve even stronger performance with stronger off-the-shelf reward model (Dong et al., 2024).
H.2. Table 3
We conduct experiments on both Gemma2-2b-ir (Team et al., 2024b) and Mistral-7b-it (Jiang et al., 2023a). Following Tunstall et and Dong et (2024), we perform training on UltraFeedback dataset iterations and show the performance of the final model checkpoint. We use the pretrained reward model from Dong al. (2024). The learning rate is set as 5c-7 and train the LLM for 2 epochs per iteration.
For the pairwise objective, we generate 2 responses for each prompt and construct the preference pair data with the reward model. For the others, we generate 4 responses per prompt and rank them with the reward model. For the contrastive objective, we construct the 1-vs-N data with the top-1 ranked response and the other responses. For the listMLE and lambdarank objective, we take the t as positives and the last-2 as the negatives. Experiments with opensource LLM as the evaluator (a ppaca eval 11ama3 70b fn) can be found Table 7.
18

---

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
80
75
Win Rate (%)
70
LC winrate Winrate
60
0.80 0.85 0.90 0.95 1.00 1.05
1.10 1.15 1.20
# Temperature
Figure Training temperature study with puir on Gemma2-2b-it and Alpaca Eval 2. Within specific range (> 0.9), lower temperature leads harder negative and benefit the trained LLM. However, temperature lower than this range cause preferred and rejected responses non-distinguishable and lead to degrade training.
H.3. Table 4
We adopt Gemma2-2b-a as the initial model. All the models are trained with iterative DPO for 3 iterations. We use the off-the-shelf reward model (Dong et al., 2024). We generate 2 responses for each prompt in each iteration. For "w. current", we only use the scored responses in the current iteration for preference optimization data construction. For "w. current + prev", we rank the responses in the current iteration and the previous one iteration, and construct the preference pair data with the top-1 and bottom-1 ranked responses. For "w. current + all prev", we rank all the responses for the prompt the current and previous iterations and construct the preference pair data. For "single temperature". and generate 2 responses for reward model scoring. For "diverse temperature", and 0.5 respective and rank the 4 responses to construct the preference data with the reward model.
only adopt temperature I generate 2 responses with temperature 1
H.4. Table 6
We use mistral- (Jiang al., 2023a) the initial model to alleviate the influence of the math related post-training data of the original model. For SFT, we conduct training on the meta-math dataset (Yu et al., 2023). For DPO, we use the prompts in the training set of the two dataset and conduct online iterative preference optimization with the binary rule-based reward (measure if the final answer is correct or not with string match). The evaluation is performed on the test set of MATH and GSMSK respectively. For SFT, we follow the same training setting with Yu et al. (2023). For DPO, we search the learning rate in 1e-7, 2e-7, 5e-7, 2e-8, 5e-8 and train the LLM for 5 iterations with early stop (1 epoch per iteration for MATH and 2 epoch per iteration for GSM8K). The learning rate is set 1e-7 and we select the checkpoint after the first and fourth iteration for GSM8K and MATH respectively.
H.S. Figure 4(a)
We conduct training with the prompts in the training set of GSM8K and perform evaluation on GSMSK testing set. We conduct learning rate search and finalize it to be 2c-7. The learning is performed for 3 iterations.
We make explanations of how we construct the four types of negative settings: For (1) a random response not related to the given prompt, we select a response for random prompt in Ultrafeedback. For (2) a response to related prompt, we pick up a response for a different prompt in the GSM8K training set. For (3) an incorrect response to the given prompt with high temperature, we select the temperature to be 1. For (4) an incorrect response to the given prompt with low temperature, we select the temperature to be 0.7.
H.6. Figure 4(b)
We conduct experiments on both Gemma2-2b-it and Mistral-7B-it models. For both LLMs, we conduct iterative DPO for iterations and report the performance of the final model. We perform evaluation on Alpaca Eval2 with
19

---

LLM Alignment Retriever Optimization: An Information Retrieval Perspective
alppca.evalllaa ama 3 ama
the evaluator.
For temperature study, we find that under specific temperature threshold, repeatedly generated responses will be large identical for all LLMs and cannot be used to construct preference data, while the threshold varies for different LLMs. The "low" and "high" refer to the value of those selected temperatures. We also conduct experiments on Gemma2-2b model and show the results in Figure 6.
H.7. Figure 4(c)
We adopt Mistral-7b-it as the initial LLM and the contrastive objective (Eq. 9) in iterative preference optimization. We generate 4/6/8/10 / responses with the L.M and score the responses with the off-the-shelf reward model (Dong et al., 2024). The top-1 scored response is adopted as the positive response and the other responses are treated as the negative responses to construct the I-vs-N training data. The temperature set as to generate the responses.
20


Reviews:
[{"review_id": "dnre9Ul6Aq", "reviewer": "Reviewer_g1d5", "summary": "The paper established a connection between LLM alignment and IR, particularly the retriever-reranker framework. With such a connection, the paper applied multiple techniques used in IR for LLM alignment, specifically, (1) IR objectives; (2) use of hard negatives (from a reasonably good model); (3) candidate list construction. Under these techniques, the paper proposed LARPO, which aims to iteratively optimize an LLM in a direct preference optimization way, so that it can learn to rank its generation responses according to a given reward model.  The paper was backed with extensive experiments against different baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback, which has significantly strengthened our manuscript. We address each comment below.\n\n- **Without hard negatives, how's the performance?** Thank you for the question. As shown in Section 6.2 and Figure 4(a), removing hard negatives (i.e., using only easy/easiest ones) leads to significantly worse performance. This highlights the importance of hard negatives in enhancing LLM capabilities.\n\n- **Without inclusiveness/memorization, how's the performance?** Thank you for the question. We have conducted a detailed study of the inclusiveness and memorization in Figure 4(c) and Table 4. From Figure 4(c), an increase in the candidate list size can contribute to improved performance (inclusiveness). From Table 4, incorporating previous iteration responses can further enhance the LLM’s capability (memorization).\n\n- **How does the number of generated responses affect the performance?** The study of the number of generated responses can be found in Figure 4(c) and also shown below. We find that increasing the number of responses improves the win rate, indicating that our method can utilize and benefit from more diverse candidates:\n\n| Length | LC Winrate (%) | Winrate (%) |\n|--|--|--|\n| 2 | 49.75 | 55.07 |\n| 4 | 50.02 | 61.72 |\n| 6 | 52.56 | 63.59 |\n| 8 | 55.21 | 64.88 |\n| 10 | 55.52 | 64.42 |\n\n- **How does the candidate list affect training time?** We evaluate training time per iteration of LarPO (contrastive variant) with varying candidate list sizes:\n\n| Length | time (min) | Winrate (%) |\n|--|--|--|\n| 2 | 25 | 55.07 |\n| 4 | 45 | 61.72 |\n| 6 | 65 | 63.59 |\n\nWe observe that larger candidate lists incur linearly higher costs but yield substantial performance gains, allowing users to balance efficiency and effectiveness trade-off.\n\n- **Algorithm 1 line 13.** Thank you for pointing this out. The operation merges all responses from previous iterations. We will revise accordingly.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Yes, no issues observed.", "relation_to_broader_scientific_literature": "The paper discussed the connection between IR and LLM, which is novel.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is well motivated and has a great presentation to connect IR and LLM alignment, which is novel. Such a connection was not only motivated conceptually, but also inspired new algorithms for LLM equipped with existing techniques from IR. \n\nHowever, as many techniques are applied for the optimization, it would be very interesting to see some ablation studies, say:\n1. Without hard negatives, how's the performance?\n2. Without inclusiveness/memorization, how's the performance?\n3. How does the number of generated responses affect the performance?\n4. Compared with traditional methods, how would the introduction of the candidate list affect the training total time?", "other_comments_or_suggestions": "Algorithm 1 line 13, $\\mathcal D_s$ should be $\\mathcal D_i$", "questions_for_authors": "See Other Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "HnPfc8KgSm", "reviewer": "Reviewer_r2Em", "summary": "This paper demonstrates that concepts from information retrieval can be ported over to shed light on numerous aspects of language model alignment tuning, including both RLHF-type functions and the data generation methods that feed those objectives. From a more technical perspective, this paper demonstrates empirically that replacing the Bradley-Terry pairwise preferences model used in the original DPO paper with listwise preference models like LambdaRank can result and improved alignment.\n\n# Update after rebuttal\n\nI appreciate the authors' thorough reply to my questions/comments, and maintain my recommendation that the paper be accepted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback, which has strengthened our work. Please find our point-by-point responses below:\n\n- **Fig 2: Single dataset?** Thank you for the suggestion. We plan to incorporate this in the revised manuscript (e.g., using NQ for both 2(a) and 2(b)). However, note that Figure 2 is intended to highlight the analogy between retriever and LLM scaling behaviors, not to compare them directly, so using different datasets seems acceptable. We adopt NQ and GSM8K since NQ is a standard dataset for retrieval; GSM8K captures LLM reasoning capabilities.\n\n- **Fig 2: Metrics are monotonic.** We agree that Recall@N and Pass@N obviously increase with N. Our goal was to visualize similar trends in scaling behavior rather than make a strong claim. We will tone down the language in the revised manuscript.\n\n- **Fig 2: \"Suboptimal\" word choice.** We agree and will revise to replace \"suboptimal.\" \n\n- **Table 2: DPO baseline with strong RM.** Thank you. We have now included iterative DPO with a strong reward model:\n\n| Model | ApEval 2 (LC WR) | ApEval 2 (WR) | MixEval | MixEval-Hard | \n|--|--|--|--|--|\n| **Mistral-Base (7B)** |\n| Iterative DPO | 32.85 | 35.2 | 0.6825 | 0.3835 |   \n| LarPO (Contrastive) | 41.5 | 42.9 | 0.718 | 0.417 |  \n| LarPO (LambdaRank) | 35.8 | 34.1 | 0.717 | 0.431 |     \n| LarPO (ListMLE) | 36.6 | 37.8 | 0.73 | 0.423 |    \n| **Mistral-Instruct (7B)** |\n| Iterative DPO | 38.95 | 47.89 | 0.698 | 0.3965 |\n| LarPO (Contrastive) | 43 | 53.8 | 0.718 | 0.425 |\n| LarPO (LambdaRank) | 41.9 | 48.1 | 0.74 | 0.44 |\n| LarPO (ListMLE) | 39.6 | 48.1 | 0.717 | 0.397 |\n\nLarPO consistently outperforms iterative DPO, validating its effectiveness with the strong reward model as well.\n\n- **Sec 6.1 vs. Table 2.** Section 6.1 isolates the effect of objectives under a fixed setup (e.g., hard negatives, candidate list construction) [1]. Table 2 reflects the joint effects of all factors. We’ll clarify this distinction in the revision.\n\n- **Fig 4a: Why GSM/Mathstral/iterative DPO?** We use GSM8K and Mathstral because such math problems have ground-truth labels, enabling clear negative type classification. Iterative DPO ensures a fixed candidate list to isolate the impact of negative difficulty.\n\n- **Fig 4b: Temperature vs. win rate.** We adjust the temperature to induce response hardness but agree other latent factors may play a role. We'll revise the explanation to focus on the importance of choosing an appropriate temperature rather than strong claims (“lower temperatures generate harder negatives”).\n\n- **Eq. 1–3: Missing document in set notation.** Thank you - we’ll update the notation accordingly with your suggestion (b).\n\n- **Fig 1: Analogy only partial.** Agreed. We’ll revise to clarify this is a similarity, not an identity. For later tokens, the prompt + decoded tokens act like a dynamic query in bi-encoder retrieval—but they differ fundamentally.\n\n- **L131–136: Embedding assumption.** Yes, we assume vocab embeddings and LLM hidden states share the same dimension, as in most LLMs. We’ll clarify this in the revision.\n\n- **r used for both re-ranker and rule.** Thank you - we’ll change the \"rule\" notation from r() to c() for clarity.\n\n- **L255: Fig 2b does not support the statement.** Thank you for the comment. The intended intuition is that lower temperatures yield more similar generated responses, increasing overlap between positive and negative samples. This effectively makes the negatives harder. We will revise the explanation to clarify this point in the revision.\n\n- **Correlation: reranker vs. alignment.** Great point. We believe a correlation may exist, but detailed exploration is beyond this paper’s scope and is an exciting direction for future work.\n\n\n[1] RLHF Workflow: From Reward Modeling to Online RLHF. TMLR 2024.", "claims_and_evidence": "Largely yes. The theoretical discussion (mapping ideas from information retrieval onto LLM alignment, both conceptually and mathematically) is sound, and the experiments are for the most part appropriately designed to measure natural axes of variation motivated by the theoretical discussion. I have the following specific comments/concerns:\n\nFig 2/Section 2.4: I appreciate the analogy. However:\n\n- I wonder if the experiment could have been done on a single data set (of course 2 different models is unavoidable). E.g. make all answers in the dataset the corpus the retriever is retrieving from.\n- Both Recall@N and Pass@N are metrics that obviously monotonically increase with N. So this experiment provides limited support for the analogy between retrieval and LLM inference.\n- \"Greedy decoding, equivalent to N = 1, is a prevalent LLM inference strategy. However, as shown in Figure 2(b), Pass@1 is often suboptimal\". I don't agree with the word choice \"suboptimal\" since in many real life scenarios you only get to attempt to answer once. And the Pass@N metric just checks whether one of your N attempts was correct, and includes no mechanism for picking which of the N samples to submit as final answer.\n\n**Table 2**: it's a nice sanity check that using a better reward model improves the performance of LARPO, however, the absence of comparison against any baselines (in particular DPO) using the better reward model makes the bottom of this table difficult to interpret.\n\n**Section 6.1**: the observations that list MLE and lambda rank improve over pairwise and contrastive objectives do not seem to transfer to table 2. At least some comment on this would be warranted.\n\n**Figure 4**: could you please comment on why GSM, Mathstral and iterative DPO are used in figure 4a (as they differ from the models/datasets/methods appearing elsewhere in the section)? I do not concur with the analysis of 4b. First, it compares temperature with win rate, not with example hardness. While it is plausible that example hardness is a latent factor and that the impact of temperature and win rate occurs *via* hardness, the experiment and figure simply don't show this. Moreover, when I see this figure I reach the reasonable conclusion that \"oh, the simple choice of temperature 1 works best.\" The analysis \"within a specific range, lower\ntemperatures generate harder negatives\" somehow presupposes that the range under consideration is [1,1.2], even though that's not even the entire $x$ axis of the figure.", "methods_and_evaluation_criteria": "Yes, although as mentioned in the claims and evidence section of this review, I have some questions around consistency in models, datasets and optimization algorithms across the various experiments of the paper. \n\nNote: I'm not asking for more experiments during the rebuttal period. Further clarification in the paper around choice of model and data set and optimization algorithm where they differ from other experiments in the paper would be adequate.", "theoretical_claims": "No, I did not closely read the appendix. However, having read the DPO paper and the first few sections of this one, I readily believe the results linking ranking objective functions to their alignment-objective analogues.", "experimental_designs_or_analyses": "See my comments on claims and evidence.", "relation_to_broader_scientific_literature": "The contributions are highly relevant to language model alignment (potentially also to language model tuning more broadly, i.e. for purposes other than being pleasant to chat with). It's possible they also have impact in applications of language models for information retrieval (by showing LLM alignment research has overlap with LLM ranking research).", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "None beyond those discussed above.", "other_comments_or_suggestions": "Equations 1-3: I get what they're trying to say, but it appears odd that the document $d$ does not appear on the right hand side of the set notation. May I suggest either of a) a pseudo-code type `sorted(D, r(q,-))[:k]` or b) briefly defining an $\\text{argmax}_k(r(q,\\cdot), D)$ that returns the function inputs $d\\in D$ resulting in top k values of $r(q,d)$ and then using that.\n\n>This analogy is further supported by the LLMs’ architecture. As illustrated in Figure 1, the generative modeling with LLMs can be interpreted as the matching process of a bi-encoder model ...\n\nThis analogy seems fair for the first generated token, but not for subsequent tokens, where the last layer representation being decoded depends on both the query and previously generated tokens of $y$.\n\nIn the equation on L131-136, is there an implicit assumption that the embedding and the embedding matrixes of the LLM are transposes of each other?\n\nL154: since $r$ is already being used to denote re-ranker, it would be best to use something other than that for the \"rule\". For that matter $f_{\\text{rerank}}$ could maybe just be $r$.\n\nL255: it's not obvious to me as a reader how figure 2b supports this statement.\n\nL301: \"More details of how the temperature \" typo.", "questions_for_authors": "Have the ranking objectives (contrastive, list MLE, lambda rank) considered in this paper, been used for the direct purpose of training LLM based re-rankers (I saw RankT5 in the related work, but it doesn't cover all of the ranking objectives in this paper and the underlying models are somewhat different)? If so, is their a correlation between the following?\n\n- the performance observed using a ranking objective for LLM-based reranker training\n- the performance observed using a ranking objective for alignment tuning", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XwkbWuJxjw", "reviewer": "Reviewer_bKqw", "summary": "This paper views LLM alignment as a retriever optimization problem, presenting a systematic framework that connects LLM alignment with information retrieval (IR) methodologies. The paper maps LLM generation and reward models to the retriever-reranker paradigm in IR. Based on three key IR principles—retriever optimization objectives, hard negative mining, and candidate list construction—the authors propose a novel alignment method, LLM Alignment as Retriever Preference Optimization (LARPO). Experimental results on AlpacaEval2 and MixEval-Hard validate the effectiveness of the proposed LARPO approach.\n\n## update after rebuttal\n\nThanks for the author's rebuttal and additional experiments. I appreciate the authors' conceptualization of LLM tuning as retriever optimization, where SFT is treated as direct retriever optimization and preference optimization is framed as reranker-retriever distillation. While this perspective is intuitive, the paper lacks a clear and convincing explanation of why IR principles are effective for LLM alignment. Unfortunately, this key concern was not addressed in the rebuttal. **I hope the authors can consider discussing this aspect more thoroughly in future revisions. I will raise my score to a 3, but I still retain this concern.**", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and believe it has significantly strengthened our manuscript. We have carefully addressed each of your comments as detailed below:\n\n- **Use of SimPO Datasets.** We exclude MTBench and Arena-Hard because (1) SimPO shows minimal differences among methods on MTBench, and (2) Arena-Hard evaluation is costly ($30+ per run). In this rebuttal, we include both below using LLaMA-3 (8B) and Mistral (7B):\n\nLlama-3-8B models\n|Model|Arena-Hard|MTBench| \n|-|-|-|\n|RRHF|26.5|7.9|\n|SLiC-HF|26.2|8.1|\n|DPO|32.6|8.0|\n|IPO|30.5|8.3|\n|CPO|28.8|8.0|\n|KTO|26.4|8.2|\n|RDPO|33.1|8.0|\n|SimPO|33.8|8.0|\n|LarPO (Contrastive)|35.2|8.3|\n|LarPO (LambdaRank)|**36.5**|**8.4**|\n|LarPO (ListMLE)|35.6|8.3|\n\nMistral-7B models\n|Model|Arena-Hard|MTBench|\n|-|-|-|\n|RRHF|5.8|6.7|\n|SLiC-HF|7.3|**7.4**|\n|DPO|10.4|7.3|\n|IPO|7.5|7.2|\n|CPO|6.9|6.8|\n|KTO|5.6|7.0|\n|RDPO|8.0|**7.4**|\n|SimPO|16.6|7.3|\n|LarPO (Contrastive)|15.4|7.2|\n|LarPO (LambdaRank)|**19.7**|6.9|\n|LarPO (ListMLE)|14.2|7.3|\n\nThese results show LarPO performs comparable or better than competitive baselines on these benchmarks. We will include these results in the revised manuscript.\n\n- **Computational Complexity.** (1) Pairwise Ranking: Computes preference between a single pair of responses per prompt, with complexity O(1). (2) Contrastive Ranking: Involves a softmax over a candidate list of size k, with complexity O(k). In practice, good performance is achieved with a small k (e.g., with k = 4, contrastive ranking performs better alpaca_eval2 win rate than pairwise ranking), making it effective. (3) LambdaRank: Based on pairwise comparisons with position-aware weighting. Its worst-case complexity is O(k²) but can be reduced to O(k) via subsampling. (4) ListMLE: Computes the likelihood of a full ranking using sequential softmax, with O(k²) complexity. This can also be lowered to O(k) using subsampling strategies.\n\n- **Comparison to LiPO[1], DRPO[2], MPPO[3], and reward margin[4][5] papers.** We would like to first emphasize that our goal is to connect IR and alignment, not to propose new listwise or hard negative methods. However, we are happy to compare our method to the five mentioned papers and add discussion accordingly in our revision. (1) LiPO [1] proposes LLM alignment as listwise learning to rank and proposes a LambdaRank-based solution. LiPO is a special case of LarPO with LambdaRank. However, we provide extensive theoretical background and demonstrate that many other ranking assumptions can be adopted in LarPO in addition to LambdaRank. (2) We compare LarPO with [2], [4], and [5] in the table below, where LarPO consistently outperforms all methods. Specifically: [2] performs well on the HH dataset (as shown in the original paper) but shows limited effectiveness on Alpaca Eval 2 and MixEval. [4] introduces a reward margin coefficient, which can misguide the LLM if the reward scores lack sufficient granularity. [5] relies on optimizing Eq. 6, but we find its training to be unstable and less robust in our experiments. (3) MPPO [3] is concurrent (posted one month before the ICML deadline), and comparison is left to future work.\n\n|Model|Ap2 (LC WR)|Ap2 (WR)|MixEval|MixEval-Hard|\n|-|-|-|-|-|\n|[2]|9.00|5.11|0.6035|0.2865|\n|[4]|13.48|10.45|0.6785|0.3395|\n|[5]|9.34|6.21|0.6360|0.3285|\n|LarPO (Contrastive)|41.50|42.90|0.7180|0.4170|\n|LarPO (LambdaRank)|35.80|34.10|0.7170|0.4310|\n|LarPO (ListMLE)|36.60|37.80|0.7300|0.4230|\n\n- **Table 2 Dataset Consistency.** All baselines in Table 2 use the same UltraFeedback prompt set and PairRM reward model if needed (as in SimPO); thus, the experiments are in fair condition. To further isolate objective effectiveness, we have added results with advanced reward models (i.e., FsfairX) under the same setup, shown in the response 4 to Reviewer r2Em.\n\n- **Temperature Study (Fig 4b).** We previously started from T=0.8 due to response concerns that lower temperatures might reduce response diversity, potentially leading to lower-quality preference data. In this rebuttal, we add these results, as shown in the table below:\n\n|Temperature|Ap2 (LC Winrate)| Ap2 (Winrate)|\n|-|-|-|\n|0.2|55.47|62.74|\n|0.4|53.71|62.09|\n|0.6|55.45|62.30|\n\nSurprisingly, low temperatures yield strong performance—possibly because the lower the temperature is, the harder the negatives are—suggesting an interesting direction on in-depth temperature analysis for future investigation. We will add this discussion to the revised manuscript.\n\n- **Ablation studies.** Section 6 includes ablations on objectives, hard negatives, and candidate construction. Our work is the first to jointly explore all these components in connecting retrieval and LLM alignment which we would think are enough contributions to the main conference.\n\n- **Reproducibility.** SimPO paper checkpoints: https://huggingface.co/collections/princeton-nlp/simpo-66500741a5a066eb7d445889. We commit to releasing all code, data, and checkpoints upon acceptance.", "claims_and_evidence": "The authors conceptualize LLM tuning as retriever optimization, where supervised fine-tuning (SFT) is treated as direct retriever optimization and preference optimization is framed as reranker-retriever distillation. While this perspective is intuitive, the paper lacks a clear explanation of why IR principles are effective for LLM alignment.", "methods_and_evaluation_criteria": "1. The authors adopt most of their offline experimental results directly from SimPO but use only the AlpacaEval2 dataset, replacing Arena-Hard and MT-Bench with MixEval. It is unclear why this dataset substitution was made. Given the reliance on SimPO's experimental setup, the authors should clarify why they did not use the same datasets entirely.\n\n2. The paper lacks an analysis of the computational complexity associated with different optimization objectives.", "theoretical_claims": "The theoretical section appears to be correct.", "experimental_designs_or_analyses": "Some experimental comparisons seem unfair. I have the following concerns: \n\n**Lack of Ablation Studies**: While framing LLM alignment as retriever optimization is a valuable perspective, the proposed method combines elements of optimization objectives, hard negative mining, and candidate list construction. Each of these components has been explored in prior work, but the paper does not provide comparisons.\n\n + For listwise preference optimization, comparisons should be made against existing methods such as LiPO [1], DRPO [2], and MPPO [3].\n\n+ The baseline methods in Table 2 use different training datasets than LARPO. To fairly assess the effectiveness of the proposed preference optimization objective, Table 3 should include comparisons against other DPO-based baselines under the same data conditions. The current results do not convincingly demonstrate that the optimization objective itself is effective, as improvements may stem from external reward model filtering rather than the optimization technique.\n\n+ For hard negatives mining, comparisons should be made with existing preference pair quality assessment techniques, such as explicit reward margin [4] and implicit reward margin [5]. This would provide stronger evidence for the effectiveness of the proposed hard negatives mining approach.\n\n**Temperature Coefficient in Figure 4(b)**: The experiment starts at a temperature of 0.8. It is unclear why lower values (0–0.8) were omitted. A complete trend analysis is necessary to understand the behavior across the full range.\n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank  \n[2] Optimizing Preference Alignment with Differentiable NDCG Ranking  \n[3] MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples  \n[4] Reward difference optimization for sample reweighting in offline rlhf  \n[5] Not all preference pairs are created equal: A recipe for annotation-efficient iterative preference learning", "relation_to_broader_scientific_literature": "The authors establish a conceptual link between LLM alignment and retriever optimization, leading to the proposal of the LARPO method.", "essential_references_not_discussed": "Please see above in Experimental Designs Or Analyses about the following papers:\n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank  \n[2] Optimizing Preference Alignment with Differentiable NDCG Ranking  \n[3] MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples  \n[4] Reward difference optimization for sample reweighting in offline rlhf  \n[5] Not all preference pairs are created equal: A recipe for annotation-efficient iterative preference learning", "other_strengths_and_weaknesses": "1. The perspective introduced in this paper is valuable and contributes to a conceptual understanding of LLM alignment through the lens of retriever optimization. However, the proposed method primarily integrates existing approaches in optimization objectives, hard negatives mining, and candidate list construction, serving more as an explanatory framework rather than introducing a fundamentally new method. This raises concerns about the validity of the claimed LARPO method, particularly due to the lack of thorough ablation studies to demonstrate its effectiveness. Consequently, while I believe this is a valuable perspective paper, I question whether it is suitable for a main conference paper.\n\n2. The authors claim that the baseline checkpoints are from SimPO. Could you provide the corresponding checkpoint links? Additionally, will the author open-source the code, experimental data, and checkpoints used in this paper, which will enhance the reproducibility?", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Db4cYlcgd", "reviewer": "Reviewer_ToBQ", "summary": "This paper establishes the connections between the formulations of LLM alignment and information retrieval (IR). Inspired by this discovery, it introduces various practices in information retrieval into LLM alignment, including hard negative mining and ranking loss functions. Empirical studies demonstrate the effectiveness of the proposed approach.\n\n## update after rebuttal\n\nThe additional empirical results and conceptual explanations by the authors have mostly addressed my earlier concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback and address the points below:\n\n- **Baselines with Strong Reward Models.** We include iterative DPO with a strong reward model for fair comparison:\n\n| Model | ApEval 2 (LC WR) |  ApEval 2 (WR) | MixEval | MixEval-Hard | \n|--|--|--|--|--|\n| **Mistral-Base (7B)** |\n| Iterative DPO | 32.85 | 35.2 | 0.6825 | 0.3835 |   \n| LarPO (Contrastive) | 41.5 | 42.9 | 0.718 | 0.417 |  \n| LarPO (LambdaRank) | 35.8 | 34.1 | 0.717 | 0.431 |     \n| LarPO (ListMLE) | 36.6 | 37.8 | 0.73 | 0.423 |    \n| **Mistral-Instruct (7B)** |\n| Iterative DPO | 38.95 | 47.89 | 0.698 | 0.3965 |\n| LarPO (Contrastive) | 43 | 53.8 | 0.718  | 0.425 |\n| LarPO (LambdaRank) | 41.9 | 48.1 | 0.74 | 0.44 |\n| LarPO (ListMLE) | 39.6 | 48.1  | 0.717 | 0.397 |\n\nLarPO consistently outperforms iterative DPO under the strong reward model, demonstrating the strength of our proposed frameworks.\n\n- **Model Size Limitation.** Training larger models is resource-intensive and cannot be easily handled by our current computational devices. Our current results on 2B and 7B models show LarPO’s consistent effectiveness across these model sizes and families. We leave scaling to larger models for future work.\n\n- **LLaMA-3 Results.** We include LLaMA-3-8B model results below. LarPO achieves the best performance across metrics, outperforming all baselines including SimPO:\n\n| Model | ApEval 2 (LC WR)  | ApEval 2 (WR) | Arena-Hard | MTBench | MixEval | MixEval-Hard |\n|--|--|--|--|--|--|--|\n| SFT | 26.0 | 25.3 | 22.3 | 8.1 | 0.742 | 0.4005 |\n| RRHF | 31.3 | 28.4 | 26.5 | 7.9 | 0.743 | 0.4125 |\n| SLiC-HF | 26.9 | 27.5 | 26.2 | 8.1 | 0.752 | 0.4515 |\n| DPO | 40.3 | 37.9 | 32.6 | 8.0 | 0.7715 | 0.4675 |\n| IPO | 35.6 | 35.6 | 30.5 | 8.3 | 0.756 | 0.452 |\n| CPO | 28.9 | 32.2 | 28.8 | 8.0 | 0.7665 | 0.4225 |\n| KTO | 33.1 | 31.8 | 26.4 | 8.2 | 0.7645 | 0.461 |\n| ORPO | 28.5 | 27.4 | 25.8 | 8.0 | 0.7545 | 0.445 |\n| RDPO | 41.1 | 37.8 | 33.1 | 8.0 | 0.779 | 0.4645 |\n| SimPO | 44.7 | 40.5 | 33.8 | 8.0 | 0.732 | 0.4185 |\n| LarPO (Contrastive)| **47.72** | **49.81** | 35.2 | 8.3  | **0.7795** | 0.4555 |\n| LarPO (LambdaRank) | 46.2 | 49.07 | **36.5** | **8.4** | 0.779 | **0.4785** |\n| LarPO (ListMLE) | 44.67 | 47.51 | 35.6 | 8.3 | 0.7705 | 0.456 |\n\n\n\n- **Comparison to LiPO.** LiPO proposes a LambdaRank-based listwise objective, which can be viewed as a special case of LarPO. However, their objective designs are based on learning-to-rank heuristics without a grounded theoretical basis. In contrast, LarPO is supported by theoretical foundations and generalizes to multiple ranking objectives beyond LambdaRank. We will add this distinction in the revised manuscript.\n\n- **W1. Notations in section 2 are not self-contained.** We acknowledge this and will revise to ensure all notations are self-contained and clearly defined.", "claims_and_evidence": "The claims are mostly supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem or application at hand.", "theoretical_claims": "I checked the proofs in Appendix E and did not find issues.", "experimental_designs_or_analyses": "I checked the experimental designs and analyses.\n\nPossible issues include:\n\n**I1.** The authors only compared the baseline approaches with the weaker reward model, and it is unclear how the baseline methods perform with the stronger reward model.\n\n**I2.** The experiments are restricted to small models with 2B or 7B parameters.\n\n**I3.** SimPO also studies two Llama-3 models in addition.", "relation_to_broader_scientific_literature": "To the best of my knowledge,\n\n- The IR perspective is novel.\n- The paper considers three IR-inspired objectives, including contrastive ranking and two list-wise ranking objectives (LambdaRank and ListMLE). [1] has previously explored using learning-to-rank objectives. The differences between the objectives employed by [1] and this paper are not sufficiently discussed.\n- The empirical studies follow the practice (setting, benchmarks, baselines) of existing efforts [2].\n- The mining of hard negatives in iterative alignment is novel.\n\n[1] Liu et al. LiPO: Listwise Preference Optimization through Learning-to-Rank.\n\n[2] Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "### Other Strengths\n\n**S1.** The paper is well-written and easy to follow overall.\n\n### Other Weaknesses\n\n**W1.** Notations in section 2 are not self-contained.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper makes an interesting connection between LM alignment and the retrieval-reranker IR framework. This allows the authors to propose a new method they call LARPO. It opens a range of new tricks for alignment including using hard negatives and candidate lists.

The paper is very well written, novel and connects two bodies of literature in a creative way that will enable more cross-proliferation and transfer of techniques. Experimentally the authors show good results on AlpacaEval2 and MixEval and compare to numerous prior techniques like DPO, SimPO and several others. 
The authors also did a solid job in responding to reviewer questions by providing additional experiments to address comments and concerns. Overall a solid contribution.
Decision: Accept (poster)