Eliminating Position Bias of Language Models: A Mechanistic Approach
Authors: Ziqi Wang
Hanlin Zhang
Xiner Li
Kuan-Hao Huang
Chi Han
Shuiwang Ji
Sham M. Kakade
Hao Peng
Heng Ji
Venue: ICLR 2025 Poster (2025)
Keywords: Position Bias
Languague Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a method to eliminate the position bias in LMs with a training-free zero-shot approach and therefore improve LMs performance.

Abstract: Position bias has proven to be a prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given context. This bias often leads to unexpected model failures and hurts performance, robustness, and reliability across various applications. A simple mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and position embedding. Based on the analyses, we propose to **eliminate** position bias (e.g., different retrieved documents' orders in QA affect performance) with a **training-free zero-shot** approach. Our method changes the causal attention to bidirectional attention between documents and utilizes model attention values to decide the relative orders of documents instead of using the order provided in input prompts, therefore enabling Position-INvariant inferencE (PINE) at the document level. By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning. Notably, PINE is especially useful when adapting LMs for evaluating reasoning pairs: it consistently provides $8$ to $10$ percentage points performance gains, making Llama-3-70B-Instruct perform even better than GPT-4-0125-preview and GPT-4o-2024-08-06 on the RewardBench reasoning set.

Published as a conference paper at ICLR 2025
ELIMINATING POSITION BIAS OF LANGUAGE MODELS: A MECHANISTIC APPROACH
Ziqi Wang!
Hanlin Zhang
Xiner Li
Kuan-Hao Huang!.s
Chi Han
Shuiwang J
Sham M. Kakade²
Hao Peng
Heng Ji
ABSTRACT
Position bias has proven to be prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given con- text. This bias often leads unexpected model failures and hurts performance, robustness, and reliability across various applications. A simple mechanistic analysis attributes the position bias two components employed in nearly all state-of-the-art LMs: causal attention and position embedding. Based on the analyses, we propose to eliminate position bias (e.g.. different retrieved doc- uments' orders in QA affect performance) with training-free zero-shot ap- proach. Our method changes the causal attention to bidirectional attention between documents and utilizes model attention values decide the relative orders of documents instead of using the order provided in input prompts, therefore en- abling Position-INvariie position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmentedd QA, molecule generation, and math reasoning. Notably, PINE is especially useful when adapting LMs for evaluating reasoning pairs: consistently provides formance gains, making ala 3-700 Instruct perform even better than GPT-4-01 1:2.-prreviiee ew and GPT-4o 2024 06 on the RewardBench reasoning set
inferencE (PINE) the document level. By eliminating
10 percentage points per-
INTRODUCTION
Language models (LMs) (Brown al. 2020
Chowdhery et al. 2022:
Touvron al.
2023 Achiam
et al. 2023) demonstrate impressive performance in general language tasks such as dialogue Thoppi
lan al. 2022), reasoning (Chowdhery al.) 2022). and schema induction Li et al. (2023). However,
they tend to favor content at certain positions (Zheng et al.
a024b|a; Wang et al. [023 Dominguez (0244) which harms complex
Olmedo et al. 2023 Zhu al., 20233 Chen et al. 2024b Liu
reasoning (Cen et al. 2024b), long-contex: understanding Liu et al. 2024) and model-based evalua-
tion (Zheng et al. 2024b). ). For example. LMs tend favor the first when is required to compare the quality of two candidate responses (Zheng et al. (024b), which hurts their reliability when being used evaluators (Figure upper)); vision-language models perform better in the recognition when the target content is presented at the bottom of the image (Figure lower right, see more examples in Appendix A). Different from ad hoc solutions from previous works (Ratner et al.) 2023 Cai et al.. 2023 Hao al. 2022 Junqing et al. 2023) Zhu et al.) 2023), we seek to understand the causes of position bias and propose to eliminate (not just mitigate) the position bias without training and searching.
We start by analyzing the key components of tfate-of-the-at LMs - Casual Attention and Position Embedding. They are the key the success of Transformers (Vaswani al. 2017), and are also the only two operations in Transformers (Vaswani et al. (007)) that will bring undesirable position bias. This is because other operations do change representations when position changes (Section 3.2). Moreover, we find an interesting phenomenon through simple experiments and give our hypothesis:
University of Illinois Urhana-Champaign, Harvard University, Texas A&M University. Correspondence:
zigivo@@llinoss.edu

---

Published as a conference paper at ICLR 2025
Question: <Questions
Question: <Questions. Mhich one of the fullowing responses is more helpfut?
one the following responses nore helpful? Response <llesponee
Response 8: clesponse B>
Response <llesponse
Oulput
Response At <tesponse A
Queput
Position (Causal Attention]
Posi Fion Bias
Insert to Any Positions In the Black Bec Beckground
70
Loss
6-5
on
"Yes"
50
Question: Is there an object
45
in riqurre Answert Yei
5
15
20
Pusiton of Document with the Ansea Position Blas (Position Encoding)
Motivating examples showing how position bias affects model outputs. Upper: LMs are asked to select more helpful one from two given responses. The example shows that LMs are prone to prefer the response positioned at first. Lower Left: LMs (LLama-3-8B-In) --BBBIInsruutt presented with 20 documents answer question, with only one document (the gold-standard document) containing the correct answer. The blue curve represents normal inference, while the red curve represents inference without inter-document attention (RoPE position encodings are kept, concrete implementation is shown in the middle of Figure 3) The height change of the yellow and orange area reflects the position bias brought by causal attention and RoPE: causal attention generally favors distant content, but RoPE prefers nearby content. Lower Right: We insert a real-world image large black background image different positions and prompt VLMs (Fuyu-8B (Bavishi et al., )023)) to compute the loss on the ground truth token. We observe a consistent pattern that models have lower losses (black color) when images are presented at the bottom.
Figure 1:
the most popular Rotary Position Embedding (Su et al. 2024) is shown to have recency bias (Su et al. 2024: Peysakhovich & Lerer 2023) due to its long-form attention weight decay w.r.t. the increase of relative positions, and the causal attention forces unidirectional information propagation, enabling models to pay more attention to distant content. Figure [1] lower left shows this retrieval- -uugeettd
QA (Liu et al. 2024) experiment. The height change of the yellow area and orange area reflects the position bias of causal attention and RoPE. Since the yellow area is mostly wider at the beginning and the orange area generally becomes wider the end (except for the last data point), this shows that the causal attention generally tends to favor distant content, while RoPE generally tends to favor nearby content
Since attention and position embedding are the causes of position bias, we propose PINE that can eliminate position bias by manipulating causal attention and RoPE to attend to different content equally. Take the retrieval augmented QA (Liu et al., [222), a task requiring LMs to answer questions based on retrieved documents, for example. The orders of retrieved documents should not affect the final results. To achieve this, we make the inter-document attention bidirectional so that the attention mask will equally attend to all documents. Next, we compute importance scores between documents and use them to re-assign document positions that positions in the original inputs are discarded. We prove resulting approach enables Position-invarian inference (PINE) w.r.t. documents in rraining-freeezzrr--shot manner.
To justify the effectiveness of PINE, we select four tasks: LMMas-a-judee (Zheng et al., 2024b), which prompts LMs choose the more helpful one from two given responses to question; eetrieval-augmented question-answering (Liu et al., 2024); molecule generation based on pro- vided properties; and math reasoning. In different tasks, "documents" have different meanings: responses in LM-de-a-judge, properties in molecule generation, and conditions in math reason- ing. Notably, we find our method especially useful when LMS are used to assess reasoning pairs: PINE with L ama-3 Oo-Innn- Pee-Instrucc ruct perform even better than GPT-4-01 00-00125-0 21---peeiiee and GPT-40-2024-088 06 on the RewardBench (Lambert et al. 2024a) reasoning set.
To summarize, we:
More supporting experiments this hypothesis in Section 4.3
2

---

Published as a conference paper at ICLR 2025
We first revisit the causes of position bias in transformers: causal attention and position encoding (Section (..2), and then propose a training-free approach dubbed PINE that can eliminate (with proof) the position bias given documents presumed to be position-invarian (Section 3.3
Four popular tasks across the general domain expert domains (chemistry and math) show PINE can bring performance gains consistently across different models and sizes.
RELATED WORK
Position Bias in LMs. Position bias widely exists LMs (Zheng et al. 2024b a Wang et al. 2023
Zhu et al.) 2023 Chen et al., 2024b; Liu et al. 2024; Shi et al.) (024)). The LM-as-a-judge task offers models two candidate responses to a question and asks models to select the more helpful one. It turns out that LM has primacy bias that tends to favor the first response (Zheng et al. 2024b). Retrieval-augmente
QA asks LM to answer a question based on retrieved documents. Liu ct al. (2024) Peyssakhovich & Lerer (0023) find that LMs prone to answer correctly when the document that contains the correct answer is presented at the beginning and the end of retrieved documents. Zheng et al. (2024a) points out that models favor options at certain positions (e.g.. prefer "A") in multple-choice QA. In the in-context learning task, Sahnn et al. (2024a); Xu et al. (2024) find that the order of in-context examples affects the final performance. Recently, several papers have proposed to understand the nature of position bias through prompting (Zhang et al., (2024b and calibration (Hsich
al., 2024). Our paper analyzes the phenomenon from the mechanical perspective: the computation must be resitional-ivvarinn to eliminate position bias.
Position Bias Solutions in LMs. Mitigating position bias have been studied by many literature from
many aspects, such as data augmentation with training (Junqing et al., 2023 Zhu et al. 2023), content
sorting by attention value during inference (Peysakhovich & Lerer 2023), searching Yu et al. 2024
Adila et al. 2024), calibration (Hsieh et al.] 2024), removing position encoding (Kazemnejad
ct al. 2024 Moving one step forward, some other solutions are designed to eliminate position bias. Zheng et al. (2024a b) use permutation then average classification tasks, which will have unacceptable (kk) (A is the number of segments) computational overhead when A large. Hsich et al. (2024) assumes that the position bias and real relevance are linear combinations and propose solutions accordingly. Different from them, we aim to eliminate the position bias from the mechanical perspective without any assumption at reasonable cost. Although several existing approaches are from the mechanical perspective (Ratner et al. 2023 Cai et al. 2023 Hao et al.) 2022), they only perform well in classification tasks and fail in more general setting: language generation.
3 METHODOLOGY
We start by running an example to illustrate position bias, followed by analyzing the cause of position bias, and end with our own approach PINE.
3.1 FORMULATION
We take aeureeedl-augmeented QA as an example, where current LMs' performance may greatly suffer from position bias (Liu et al., 2024). The task requires the model answer question based on a set of given retrieved documents, where only one of them contains the correct answer. The system prompt SYS is: "Write a high-quality the given question using only the provided ded search results (some of which might be irrelevant). " Given a question Q, and three retrieved documents: Di. D2. and Dz. we can formulate several different inputs. For example, [SYS]Q]D, (D2 (P3). and (Ps]) We expect models have the same output for these inputs because D2, D3, D are position-agnostic documents: their relative order is not supposed to affect the final result. However, the current LMs answer differently when presented with these different inputs and tend to answer correctly when the document contains the answer at the beginning or at the end of all documents (Liu et al. 2024). The systematic differences of model outputs caused by relative positions of documents reflect the position bias of the model. Therefore, current LMs cannot conduct inter-document position-invarian inference, and our goal is to make the inference invariant w.r.t. relative document orders. In the rest of this section, will use this running example. However,
one-sentence answer for
3

---

Published as a conference paper at ICLR 2025
SYS
D
Ds
Decoded
2
6
3
2
3
Bidirectional
2
Position
1
6
2
3
2
3
4
Attention
2
3
4
Re-assignment
3
4
5
2
3
4
5
2
3
4
5
1
3
2
5
3
4
5
6 6
3
4
5
6
1
2
3
4
3
4
5
6
1
5
ImportancelDy. D2)> Importance1D... D0:1
Compute Importance
Importance| (D, Importance(D)
sorting
Importance( by D.) Importancel Dy D2)
Importance[Token 8, D.) Importance(Token 8, > Importance D2)
Figure 2: PINE: inter-document aosition-invariant inference via bidirectional attention. The attention matrix of the running example in Section 3.1 at the left of the figure, the orange, different blue, and green colors denote system prompts (I token), three different documents (2 tokens each) and decoded tokens (1 token), respectively. The number at (i,j) the figure, Pijs denotes the position of token j when computing the attention from query q; Therefore, P.j is equal for all i in vanilla inference. PINE enables inter-document bidirectional attention and then uses attention scores between documents to compute their importance. Then, documents sorted by importances: more important documents are placed in closer positions. The computation of "importance score" is introduced in Section 3
emphasize that "documents" have different meanings different tasks: responses in L-Mass--jjude,, properties in molecule generation, and conditions in math reasoning. Therefore, readers should be aware that the method is not just designed for a single task. the rest of the paper, we merge SYS and Q into SYS for simplicity.
3.2 CAUSAL ATTENTION AND POSITION EMBEDDING ARE THE CAUSE OF POSITION BIAS
free-forwwrd networks (FFNs), Query, Key, and Value (QKV) projections, and layer normalization in the Transformer architecture do not cause position bias, they yield the same representations regardless of document positions. Rather, the attention computation that leads to the position bias:
PE Q, poso), KPF PE(K, posk)
H Softmax Q (QPE
Lemssl V
where Q, K, V e Raxd are queries, keys, and values, PE donotes the position encoding, pos and posk denote the position of queries and keys, and claassal denotes the causal attention mask. Eq. [ reveals that (1) the PE function yields different representations for documents if their orders changes, therefore affecting the importance score QF K3 and hidden states; (2) the 1 causal generates different attention masks for the input documents if change their positions, resulting different hidden states. To achieve inter-document position- position-invariant inference, H must remain the same regardless of documents' orders.
3.3 PINE: ATTENTION
DOSITION-INVARLAE INFERENCE VIA BIDIRECTIONAL
Our goal is to obtain an inter-document position-invariant hidden state HPINE. which does not change regardless of document orders. We can mechanistically eliminate the position bias by equally attending to all documents. Therefore, we propose PINE, an approach that uses bidirectional inter- segment attention and re resisiigg positions by importance scores (computed from attention score) to eliminate position bias (Figure We address that the "elimination" and "invariance" in our method talked about from the input-output perspective, i.e., outputs remain unchanged regardless of the
4

---

Published as a conference paper at ICLR 2025
input-position orders. PINE still uses position encoding and does not eliminate position encoding itself.
Bidirectional Attention. We first change the attention mask so that documents can attend to each other. Specifically, we make the inter-document attention bidirectional but keep the intra-document attention causal (Figure middle). Our goal is to eliminate "inter" position bias among different documents rather than "intra" position bias within each document. The latter will lose the order information of tokens, and models can degenerate into bag-of-words models, which is not what we expect.
Re-assign Positions: Sorting By Importance Scores. Re-assigning positions must consider two folds: the position of queries and keys. Each token in conventional LMs has the same position when serving as both query and key. In the bidirectional attention we use, this assignment has be reconsidered. First, LMs are trained causally, meaning the position of the query must be larger than the keys in the attention computation. Therefore, it necessary to manipulate positions so that each document is the last document when serving as queries (the diagonal of the rightmost figure in Figure 2). For tokens before and after documents, their positions are not affected when serving as queries.
Re-assigning positions for keys must be redesigned to eliminate position bias. We determine the positions of documents based on importance scores when they serve as keys (numbers in the rightmost part of Figure 2). Specifically, we first compute the attentions without position embedding involved: Importance (i Softmax(q:k3 /Vd), where d is the hidden state dimension. Then, we obtain the importance score between documents by aggregation. For example, Importance D2) DieDijeD, (tportanceeskemn..))) ( The length normalization is to prevent assigning higher importance scores to longer documents The importance score could also be computed between individual tokens (e.g., Token 8) and documents. Lastly, we re-assign positions by importance scores as shown in the rightmost part of Figure 2 more important documents will have closer positions to the query. The rightmost part of Figure shows the concrete position re-assignment for keys (its diagonal also represents the position re-assignment for queries). To avoid confusion, we address the fact that we do not actually sort tokens and only re-assign them to different positions. In our position re-assignment, the position of keys may vary depending on the queries (numbers in column are different), which is the key difference between PINE and vanilla inference. Besides, our method not limited specific position embedding types.
Inter- mnter-Document Position Invariant Inference. Once we have new attention mask and position re-assignment, we can place them into Equation I and obtain HPINE. By applying HPINE to every layer, attention heads, and tokens, we reach our method PINE. We prove that:
Lemma 1. the input Q.K. V are dnter-document masition-imwarian representations, then HINE are also inter-document position- position-invariant invariant representations.
Proof: To simplify the notation and without loss of generality (w.l.o.g), we still use examples in Section 3.1
First, the SYS tokens already satisfy this lemma under the vanilla inference since they appear before documents, and PINE does not change their computation process. We only need to show PINE can make D; and Token 8 (i.e., tokens after documents) satisfy the lemma. W.l.o.g, we use Dj as a running example:
PINE first obtains importance score between documents: Sim D1, Dz) Σ Softmax (Q, (/Kaan/XXK/////////.... where Q1 E 2x2dd K; Rx2dd 2 denotes the num- ber of tokens in documents, and d denotes hidden states dimensions. Note that here the Q, K have not been applied to position embedding yet. Therefore, the importance score is not function of input document positions.
W.l.o.g, g, let's assume (im, D2) > Sim(D1, Da), then we sort the document as follows [Da] /D2 [P[i] when they serve as keys and D, as query. Concretely, QPE,1 PE(Q,, 3) (3 de- notes is treated as the last, i.e., third, document), KPE,1 = PE(K,, 3), KPE,2 PF(K2, 2),
'In our pilot experiments, we find summation makes models convert from position bias to length bias. We also try maximum instead of averaging and find this methods usually have noticeably worse performance than averaging possibly due to noises brought by unimportant tokens.
5

---

Published as a conference paper at ICLR 2025
SYS
D3
Decoded
2
2
3
Attention Masking
3
Equal Position
1
2
3
4
1
2
3
4
5
4
1
2
3
4
5
6
2
3
4
5
6
7
2
3
4
5
6
7
8
2 3
7
6 8
2023 eliminates position bias by first masking all inter-document attention and then re-assigning all documents the same positions. The notions are kept the same as Figure 2 Our experiment Section shows that PCW brings severe performance drop for tasks requiring language generation.
Figure 3: Previous work PCW (Ratner al.
KPE,3 PE((3, 1). Then compute hidden states of D1: H1 Softmax DerimaxxQQeee (QE, KPE/ vd), where Kpg is the key values for the whole sequence (S process does not use any variables that are dependent on the input document positions, nor directly use the input document positions. Therefore, H obtained by PINE is not function input document positions.
1 It noted that this
Similarly, H2, H3, and Token 8's hidden states are not functions of imput document positions.
Their concatenation yields HPINE, which is not function of input document positions.
Proof ends.
Theorem 1. Given an input, if HPINE is applied to every layer. attention head, and token to replace the conventional attention computation, then the model outputs inter-document position- invariinttnv representations.
The theorem can be proved by mathematical induction by (1) lemma, (2) FFN, QKV projection, and layer norm yield representations that are not function of document positions, and (3) the embedding representation is not function of document positions. We put the complete proof in Appendix B This lemma can also be understood in simpler way: it is corollary of the symmetry principle Gross (996).
Some takeaways that are worth noting: (1) Both bidirectional attention mask and position re- assignment are needed to complete the proof. (2) PINE needs to applied to every layer, attention heads, and tokens to complete the proof. (3) PINE is not limited to specific position embedding types.
3.4 DISCUSSION
Different Position Re-Assignment Methods. PINE puts documents with higher importance scores closer position queries. Another option is put documents with higher importance scores in more distant position to the queries. Considering the recency bias brought by the most popular rotary position embedding (RoPE) (Su 2024), this alternative approach makes RoPE "disrespect" the attention of models. Therefore, we believe this alternative choice is not optimal, which is justified by our experiments in Section
Different Attention Masks. Previous work PCW (Ratner et al. 2023) adopts different way: it masks the inter-document attention instead of making bidirectional (Figure middle and right). Accordingly, it adopts simplified position re-assignment method of ours: putting all documents in the same positions. However, masking all inter-document attention loses contextual information (the white part surrounded by colored blocks in Figure B}. Moreover, some different tokens share the same positions (Figure right), which could confuse models. As a result, PCW performs poorly in language generation tasks (Section 4
Inference Cost. PINE incurrs additional computation overhead due to extra operations. Practically, the extra big O computation complexity to obtain hidden states is O(nk log k), where and denote text length and the number of input documents, respectively. The bidirectional attention does not bring extra cost, the position re-assignment brings Ol log k)) for each token since the sorting

---

Published as a conference paper at ICLR 2025
algorithms are involved. The real computation cost is acceptable since it is usually small (e.g., k in the LLM-as-a-juue judge task and 20 in the retrieval- auggeneed QA). Section 4.5 shows results of real-world wall time and memory cost.
EXPERIMENT
Our experiments aim to show PINE can improve model performance across diverse tasks and have superior performance than other approaches.
4.1 SETTINGS
We select four tasks that pose position bias: LM-as-a-judge (Zheng et al., 2024b) that prompts LMs to select better response out of two given question, retrieval-augmennte
preestion-asswens (Liu ct al. 2024) that asks LMs to answer questions based on retrieved documents, molecule generation based on provided properties (Ramakrishnan et al. 2014), and math reasoning based several given
conditions Chen et al. (2024b). We follow previous work (Liu et al. 2024 Lambert et al.
2024a and
temperature in avoid variance.
LM-as-a-judge- We benchmark our method on 23 datasets in the RewardBenet Lambert et al. 2024b) that can be categorized into four types: Chat, Chat-Hard, Safety, and Reasoning. We the official data split, prompts, and evaluation scripts to ensure reproducibility. We use LLaMa-3-Instrut models (AI 2024) and Swen-115-Chin models (Bai al., 2023) for experiments. To show how positions affect results, we present four results: the ground-trun trrut response is positioned at first, second, or shuffled, and PINE results (which yield the same results for all three scenarios above).
QA. We follow the settings and use the prompts, data, and evaluation scripts of (Liu et al. 2024| Only one of the retrieved documents (10 or 20 in total) contains the ground-true answer for the given question. We list prompts in Appendix E We use Ba-Maa---7BBInstruee model (AI, 2024) for experiment. To show how positions affect results, we present several results: the ground-truth document is positioned at the beginning, middle, last, or shuffled, and PINE results (which yield the same results for all scenarios above).
Retrieval-augmentt
Molecule Generation and Math reasoning. We also conduct two bonus experiments. Molecule generation based on given properties (property positions can be swapped), and math reasoning where conditions can be swapped.
More details the four tasks can be found in Appendix Qualitative examples of the four tasks can be found in Appendix [F]
Baselines. The goal of PINE to climinate position bias during inference mechanically. Therefore, choose methods that have the same design principle our baselines: (1) Vanilla inference (2)
Vanilla inference with no inter-document attention (NIA for short, i.e., the middle figure Figure.
3) The latter documents will have attention formers. (3) Parallel Context Window (PCW, rightmost in Figure. 3 [Ratner et al. 2023): PCW extends the baseline (2) by manipulating positions of documents. PCW allows all documents to share the same positions. (4) Structured Prompting (SP, a variant version of PCW) Hao et al. (202)) SP extends (3) by lowering attentions between decoded tokens and input documents
& to solve the perplexity exploding problem in PCW. Similar the proof in Section 3.3 we can know that (1) and (2) are not inter-document position invariant, whereas (3) and (4) are. Beyond these methods, we also introduce two other debiasing baselines: permutation
(Zheng al. 2024a) and calibration (Zhao et al. 2021).
4.2 RESULTS ON
Position bias exists across different models and sizes. We first analyze the statistics of position bias in RewardBench with different models (Appendix C.. Position bias is quite common in RewardBench, and can be up to 48. 0% Larger models have less position bias, however, the position bias could still on average affect up to 10% data.
"Apache-2.0 license. https///ithuhh.ommmlllcm/aaarrre MITT license. http://ghhi....commmoeeoriiiiltt
7

---

Published as a conference paper at ICLR 2025
Table 1: Main results of RewardBBnnch. Vanilla denotes the normal inference, (GT A) means the ground truth chosen response is presented at the first, and (GT at B) indicates the second. For the 72B model, we additionally benchmark the Qwen 2.5 model. PINE consistently improves LM's performance across different models and sizes and is particularly useful when assessing reasoning pairs.
Llana-3
Instruct
Qwen-1.5 -.S-hal
Method
8B
70B
18B
4B
7B
32B
72B / 72B (Qwen 2.5)
110B
RewardBench (Full set)
Vanilla at A)
67.5
78.0
36.3
29.5
61.4
74.2
79.6/8872
Vanilla (GT at
06.3
76.5
66.2
76.6
59.6
74.8
09.5/ 80.5
75.7
Vanilla (Shuffle)
64.8
76.0
50.3
53.1
60.9
72.8
72.878344
81.1
PINE
66.7-1.9
77.441.4
62.9.2.6
58 58.2.s.1
61.5+0.6
74.8.2.0
71.8-1.1
/81....
82.9...7
RewardBench (Reasooning set)
Vanilla (GT at A)
80.3
87.8
43.3
42.8
62.1
78.3
83.0/93.7
90.0
Vanilla at
66.0
80.3
62.3
54.3
73.6
78.0
Vanilla (Shuffle)
65.3
78.9
48.4
54.1
59.3
66.8
68.27
78.0
PINE
73.4 s.1
87.6
60.1
11.7
61.0
6.0
63.0 a..7
76.7
19.9
69.0.
99..3
86.2,
Table 2: Baseline performance on RewardBench. PINE achieves superior performance to baseline models, performing 4.8% and 4.7% better than the best performed baseline models.
LLaMa-3-8B-Insse
Swenl.557B-Chiee
Method
Reasoning
Full Set
Reasoning
Full Set
NIA (GT at A)
43.7
56.3
60.7
61.3
NIA (GT at B)
66.7
65.8
44.1
52.2
NIA
55.9
61.9
51.4
56.8
PCW
56.5
61.7
53.4
55.2
SP
55.4
60.8
52.4
55.4
PINE
73.4 16.9
66.7
63.0
61.5
PINE consistantly improve model performance across models and sizes. Table [hoows the main results RewardBench. We experiment with Llama-3 and Qwen-1.5 across different model sizes. The position of the ground truth chosen option is randomly shuffled. Therefore, the accuracy of the random guess method is expected to be 50%. First, the first two reveal that larger models tend to have a primacy bias, whereas smaller models tend to have recency bias. By comparing the last two rows of each model size, conclude that models across different sizes perform better with the help of PINE by eliminating position bias. The only exception is the Qwen-1.5-722 1.5--2B--hChhh model. We well-trained since Dowen-1.5-32B--
suspect this model in vanilla inference, despite half of the model size. Qwen 2 report (Yang et al.) (024) also shows that the Qwen 1.5B 72B model performs even worse than 32B reasoning. Moreover, Table shows that Qwen 2.5 72B can obtain consistent performance gains. Overall, PINE improves performance from statistical perspective and makes models more reliable when as evaluators. Full results are shown in Appendix D
performs the same the 72B model
PINE extremely useful when assessing reasoning problems in RewardBench. PINE consistently improves model performance the "reasoning" subset by large margin: from 8 to 10 percentage points in most cases. Specifically, LlaMa-3 Instruct 70B was originally ranked 22nd generative model in the reasoning subset of RewardBench. With PINE, it achieves the 7th rank (87.6%), out- performing GPT-4-012 -015--peviiew (the previous 8th rank, 86.9%), -PT-402024-------------- (the previous 9th rank, 86.6%), and Llama rank, 87.1%) 6
Inwtr-IInttuutuuuuuu (the previous 7th
PINE performs better than baseline models that adopt different attention masks. We then compare PINE with haseline models mentioned in Section 4.1 on -Jama-33-8B-Instreee and Qwen 1.5-- 7B-Chat model. They adopt a different attention mask: masking inter-document attention instead of making them bi-directional. Since NIA is not inter-document position-invariant, we also apply NIA with two extreme cases: the ground truth chosen response is always in the first or second
Resslls are provided by the official leaderboard (as of Sep 17, 2024): https: htpss://uugqtt www.h//gggggaacc........ spaces/allenai htw.e/eennaeeeerdreeeeeee
8

---

Published as a conference paper at ICLR 2025
Vani la
Vanil (Shuffle)
(shurfie) SP
solgement (iltte
PINE
sosipement
80
70
78
65
70
Accuraity
60
60
74
55
72
50
40
70
45
30
68
Position of Document with
Position Document with Answer
5
10
Pocitiin of Derument with the dnewer Total document:
Total document: 10
Total document: 20
10
(a)
(b)
Figure 4: The results of retrieval-augmeentee QA on 1eiiaa---7B-IIntttee Dashed lines indicate that the method is either inter-document position-invariant or the result is obtained the order-shuffled data (denoted in the legend). (a) shows results of PINE against baselines. (b) shows results of different designs of PINE.
place. Results Table 2 show that PINE achieves the best performance and largely outperforms the best baselines by 5%, and outperforms NIA even if NIA is placed in the extreme case. On the reasoning subset, this performance gap becomes much even greater. The results reveal that masking inter-document attention mask is much less effective than bidirectional inter-document attention mask applied in PINE.
PINE performs better than permutation and calibration methods. Another two widely used debiasing methods are permutation (Zheng et al., 2024a) and calibration (Zhao al. [021]. They are usually used in the logit-based evaluation or single-token generation. Their effectiveness in the open-ended generation is less explored. In our experiments, we find calibration methods generates rubbish responses, which we believe is because of the strong assumption in (Zhao et al. 2021): uniform distribution of all tokens in the generation task. For the permutation methods, we find LLama-3-8B: -nnsstrrc have 69.0% and 65.9% accuracy, Swenl.5-7B-Chhe has 58.2% and 61.3% accuracy on the reasoning set and fullest respectively, all underperforming PINE (numbers reported in Table
4.3 RESULTS ON RETRIEVAL-
QUESTION-ANSWERR
PINE performs better than baselines, on-par with vanilla inference on average while not being affected by the worst case. Models tend to perform better when the gold standadd document is at the beginning and the end of all documents in retrieval-augmented question-answers. Figure 4 (a) shows the results on LLaMa-3-70B-Inm to conclude that all baselines are much worse than PINE (the pink line), which is consistent to the previous experiment. Second, PINE achieves on- pa-par performance on average compared with vanilla inference while being inter-document position invariant. Specifically, PINE is slightly better/worse than vanilla inference with the gap +1.2 +1.2/-2.0 when there are 10 and 20 documents in total. We hypothesize that the slight performance drop of PINE for the 20 document setting is due to the performance drop of document importance score computation in PINE when presented with many documents. However, PINE is position-invariant, therefore does not be affected by the worst case (the bottom of blue solid curves). Third, the height generally becomes smaller between blue and brown solid lines in Figure (a), and between the blue and red solid lines in Figure 4 (b) when the gold-standard document position increases, reflecting the causal attention generally prefers distant content, which consistent to the hypothesis in Section [] The brown line in Figure 4 (a) and red line (b) generally reflect recency bias brought by RoPE, which is consistent to previous works (Su ct al. 2024. Peysakhovich & Lerer. 2023).
when 10 or 20 documents were presented. First, is easy
(a),
PINE performs better than other position assignment methods. So far, experiments show that bidirectional dnter-document attention is the better design choice than the masked one. However, there still several design options for the position assignment, discussed in Section The first option is to re-assign position reversely, and the other is to use PINE without position re- re-assignment (i.c., use input document positions when they serve keys). To gain deeper understanding, extend the

---

Published as a conference paper at ICLR 2025
Table 3: The result of molecule generation QM9 dataset. PINE improves model performance in out of 6 criteria.
Model
ar
EHOMO
ÉLUMO
A
C₁
LLama
6.3997
103.93
53.4
99.13
3.4112
4.3785
Llama PINE
6.3702
102.15
53.09
98.27
3.4917
4.2886
retrieval-auue auggeetee QA experiments with the two mentioned alternative position assignment methods, and the results presented in Figure 4 (b). The figure tells that PINE is slightly better than PINE without position re-assignment on average (+0.3. The gap becomes larger when 20 documents are presented: +1.5). Position re-assignment reversely has relatively worse results, showing that PINE is a better design choice, which is consistent with the intuitive analysis mentioned in Section 3.4 Although position re-assignment seems only bring less gains than bidirectional attention mask, it is required to complete the proof that PINE can eliminate the position bias. Therefore, PINE without position re-assignment may suffice if one does not aim to eliminate the position bias and cares more about efficiency (no extra O(nk log k) sorting cost).
4.4 RESULTS MOLECULE GENERATION AND MATH REASONING
PINE improves model performance on out of 6 cri- teria in molecule generation Table 3 shows the results
of molecule generation. The consistent gain in 5 out 6 criteria shows the efffcctiveness of PINE.
Vanilla
70
PINE improves math reasoning capabilities. Figure 5 shows the results of Qwenl.5 models on R-GSM dataset. can be shown that PINE outperforms vanilla inference for both small 7B models and large 110B models.
Accurals
50
30
20
4.5 COMPUTATIONAL OVERHEAD
10
Section 3.4 briefly discusses the computational overhead, with conclusion that PINE 's efficiency is still doable. In our experiments, we find the wall time PINE is ~2x and ~8 of the vanilla inference the LM-as-a-judge task and retrieval-augmented QA task with 20 documents, which acceptable least during experiments. However, we did not specially optimize codes to accelerate PINE, and our implementation still contains "for" loop. Therefore, we believe there is room to accelerate PINE. Compared with the time overhead, the memory overhead is small and PINE can be run with 70B models on 3x A 100 80G on the setrieval-augmenn QA task, which requires the same number GPUs as the vanilla inference. Since efficiency not the main focus of this paper, we leave this our future work.
7B
1108
Figure 5: Math reasoning results of Qwenl.5 series on R-GSM subset. PINE improves the reasoning accu- racy by 12.6% and 5.3% with and 110B models respectively compared with vanilla inference.
CONCLUSION, LIMITATIONS AND FUTURE WORK
We propose a novel train-free zero-shot approach to eliminate the position bias mechanically. The core idea is to make every input documents equally affected by the attention mask and position embedding. However, PINE requires extra computation. We believe there room to improve the efficiency with more efficient implementation, and we leave this as future work.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Experiment details are described in Section 4.1 and Appendix E
Codes are uploaded to: ht
//githubs com/wzq01 6/P INE
complete proof of the lemma and theorem occurred in Section
3.3 are presented in Section 3.3 and Appendix
ACKNOWLEDGMENT
We thank Chujie Zheng for the helpful discussion and feedback. This research is based upon work supported DARPA ITM Program No. FA8650-23-C--3166 The views and conclusions contained herein those of the authors and should not be interpreted necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Dyah Adila, Shuai Zhang, Boran Han, and Yuyang Wang. Discovering bias in latent space: An unsupervised debiasing approach. arXiv preprint arXiv:2406.03631, 2024.
Meta AI. Build the future of ai with meta Ilama 3, 2024. URL http::htttt:mmmmmm www.maammtta...mm com/
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, al. Pytorch Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 929-947, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL ltttps:://wwtt adept ai/ blog/ fugg//uuu-bb
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Tianle Cai, Kaixuan Huang, Jason D. Lee, and Mengdi Wang. Scaling in-context demonstrations with structured attention. In Workshop on Efficient Systems for Foundation Models ICM1.2023, 2023. UtLhttps: httpatpsp//ppeeeeeeee..cooo net Forum? hat.rrmmmmmmm.H00008.6333 1H5802KKPM
Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in rea- soning with large language models. ArXiv, abs/2402.08939, 2024a. URL https://api semant eschol.c..... ar. wwwprrorg/ooouuuuooe
267 657940
Xinyun Chen, Ryan Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.0899. 08939, 2024b.

---

Published as a conference paper at ICLR 2025
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Weier-Hellsten Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünnnr. Questioning the survey responses of large language models. arXiv preprint arXiv:2306.07951, 2023.
David Gross. The role of symmetry in fundamental physics. Proceedings of the National Academy of Sciences, 93(25): 14266-14259, 1996.
Yaru Hao, Yutao Sun, Li Dong. Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples, arXiv preprint arXiv:2212.06711. 2022.
Cheng- Yu Hsieh, Yung- Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. Found the middle: Calibrat- ing positional attention bias improves long context utilization. arXiv preprint arXiv:24006 /6008, 2024.
He Junqing, Pan Kunhao, Dong Xiaoqun, Song Zhuoyang, Liu Yibo, Liang Yuxin, Wang Hao, Sun Qianguo, Zhang Songxin, Xie Zejian, et al. Never lost in the middle: Improving large language models via attention strengthening question answering. arXiv preprint (eXiv:2311.09190 2023.
Amirhossein Kazemmejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy, The impact of positional encoding length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling. 2024a.
c0/ co/spaces/alleeaiiiee http://aaaaaaaaaaaaaeeeeeeee
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprini arXiv:2403. 13787, 2024b.
Sha Li, Ruining Zhao, Manling Li. Heng Ji, Chris Callison-Burch, and Jiawei Han. Open-domain hierarchical event schema induction by incremental prompting and verification. In Proc. The 6lst Annual Meeting of the Association for Computational Linguisties (ACL2023), 2023.
Xiner Li, Limei Wang, Youzhi Luo, Carl Edwards, Shurui Gui, Yuchao Lin, Heng Ji, and Shuiwang Ji. Geometry informed tokenization of molecules for language model generation, 2024. URL
htttp:://axxvv..xxt wwtp:/iiiorr////oooooeooooo alss 2408 10120
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
OpenAl. Gpt-ov(ision) system card. 2023. URL httpa://aai.emmmnnccccooo..... semanticscholar.org.
api.
CorpusTD:26321800:
12

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Alexander Peysakhovich and Adam Lerer. Attention sorting combats recency bias in long context language models. arXiv preprint arXi:2330001427,2022
Raghunathan Ramakrishnan, Pavlo O.
Dral, Pavlo O. Dral, Matthias Rupp. and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014. Uttpttss: swwp:////..p..mmaa....moamam
semant: stt.appisslemmmmttcccccccoloeeee hat.:aarroorrrooorrrii111555501.
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6383 6402, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.118633vv// aat.:///////22allllllo acl-aong.-3-3o55 Io22.352. 332.. URL ht hww.h:ttps/eccccm peattps:/acclateee http://aaaaaaaaooo.....oo wwrpooy..rgg/22333 acl -long
352
Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling, E(n) equivariant graph neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learming Research, pp. 9323-9332. PMLR, 18-24 Jul 2021. URL
satorrasZla html
Lin Shi, Weicheng Ma, and Soroush Vosoughi. Judging the judges: systematic investigation of position bias in pairwise comparative assessments by Ilms. arXiv preprint arXiv:2406.077 077791 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 168::27063, 2024.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- -zze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint Stti::2001002333 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. Primacy effect of ChatGPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 108-115, Singapore, December 2023. Association for Computational Linguisties. doi: 1111866/112222.emmnnnmn www.:y..org///. 2023 http://222.ee.......mmmmmmm
URL
Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-an natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v /2200..cmmlpdddmoss..... URL https://claathhllggg org/ eww.ppdoosm.............
Penghao Wu and Saining Xie. v*: Guided visual search as a core mechanism in multimodal 1lms.
arXiv preprint 231.353/2. 2023.
13

---

Published as a conference paper at ICLR 2025
Zhichao Xu, Daniel Cohen, Bei Wang, and Vivek Srikumar. In-context example ordering guided by label distributions. arXiv preprint ar arXiv:2402.11447 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin- Lin, Dongsheng Li, Yuqing Yang. Yongfeng Huang, and Lili Qiu. Mitigate position bias in large language models via scaling single dimension. arXiv preprint arXiv:2406.0255 02536, 2024.
Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, and Rui Yan. Batch-icl: Effective, efficient, and order-agnostic in-context learning. arXiv preprint arXiv:2401.06469, 2024a.
Meiru Zhang, Zaiqiao Meng, and Nigel Collier. Attention instruction: Amplifying attention in the middle via prompting. arXiv preprint arXiv:2406. 170900 2024b.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, 2021. [tttss/s::///pi sww.at...............meemmmmaa.. cscholacc......oo Srtg///oooosst::1119710
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2024a. URL htt.ttpss//ppsneeveee..o net. wttp//orrmmmo//oom
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm-as-a as--age with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024b.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint 1XX::23310116311 2023.
14

---

Published as a conference paper at ICLR 2025
A
ANOTHER EXAMPLE OF POSITION BIAS IN VLMS
(a)
(b)
(c)
(d)
Figure 6: Image of Andromeda Galaxy. GPT-4V is asked to answer the galaxy of the redbox (M110) (b), (c), (d) images are generated by flipping (a). Results show that GPT-4V can answer correctly on (c) and (d) and incorrectly on (a) and (b), revealing position bias in VLMs.
To show VLMs suffer from position bias, we ask models to identify a small object appeared in the full image. We ask GPT-4V (OpenAl 2023 to identify one satellite galaxy M1 of Andromeda Galaxy (M31). The central objective in the 6 (a) is the Andromeda Galaxy and the galaxy in the red box is M110 satellite galaxy. Another satellite galaxy is M32 which the brightest part near the Andromeda Galaxy.
We flip the image (a) and obtains images (b), (c), and (d). We prompt GPT-4V "What galaxy in the red box?", and here are the responses we collected:
(a): The galaxy within the red box is M32, also known as Messier 32. It's a dwarf elliptical galaxy and is satellite of the Andromeda Galaxy (M31), which is the prominent galaxy the image. M32 is one of the more well-studied dwarf elliptical galaxies, largely due to its proximity to our own galaxy and its interactions with the Andromeda Galaxy.
(b): The object inside the red box in the image appears to be a
smaller galaxy located near the larger, more prominent galaxy (Andromeda Galaxy, M31). This smaller galaxy is likely M32, satellite galaxy of the Andromeda Galaxy. M32 is compact elliptical galaxy and is often visible in images as a smaller, dense, bright spot near Andromeda, as seen the boxed area of your image.
The galaxy within the red box is M110, also known as NGC 205. M110 is dwarf elliptical galaxy, which, like M32, is a satellite of the Andromeda Galaxy (M31). of the Local Group of ga galaxies, which includes the Milky Way, the Andromeda Galaxy, and about 54 other known galaxies.
(c):
is one of the members
(d): The galaxy in the red box appears to be satellite galaxy of the Andromeda Galaxy, which is the prominent galaxy visible in the rest of the image. This smaller galaxy is likely M110 (NGC 205), which is a dwarf elliptical galaxy and a companion to the Andromeda Galaxy, MB1. It's one of the several satellite galaxies gravitationall: bound to Andromeda, visible here as a faint, elongated object in the outlined area.
We can find that models answer corrected when M110 at the top of the image, revealing that VLMs also suffer from the position bias. The position bias may lead unreliable VLMs when fine-grained image analysis are needed (e.g., small object detection (Wu & Xie )023
B COMPLETE PROOF
This section provided complete proof to show PINE eliminate position bias.
15

---

Published as a conference paper at ICLR 2025
To simplify the notation and without loss of generality (w.lo.g), still use examples in Section 3.1
Theorem 1. Given input, HPINE is applied every layer, attention head, and token replace the conventional attention compulation, then the model outpuls are inter-document representations.
invariann
First, the embedding layer is not function of input documents positions. Suppose that the äth layer's input hidden states are not function of input documents positions, then within each layer:
The attention hidden states are not function of input documents positions (Lemma).
The Layernorm, FFN outputs
function of input documents positions.
Therefore, the output hidden states ith transformer layer, i.e., the input hidden states of + 1th transformer layer, are not a function of input documents positions.
Using mathematical induction, we know the final outputs are not a function of input documents positions.
Proof ends.
Notes on the proof:
PINE needs to applied on each layer, attention heads, and tokens to satisfy the above proof.
The extra big O computation (klogg)) for sorting k documents. Since we need to repeat this step for every token, the extra computation cost is Onkklog)), where n is the number of tokens.
purely come from the position re-assignment step:
Although position ee-assignment brings an extra computational cost, it a must to complete the proof. Removing this step will make PINE unable "eliminate" position bias. Similarly, a bidirectional attention mask is also a must to complete the proof.
PINE is not limited to specific position encoding algorithms.
C STATISTICS OF POSITION BIAS IN REWARDBENCH
We show the statistics of position bias in RewardBench in Table
Table 4: The portion of data (%) that models have position bias RewardBench, i.c., models change answers after swaping candidate responses orders. We color the subsets that have more than 25% data causing position bias with cyan.
Model
Size
Chat
Chat-Hard
Safety
Reasoning
Avg.
LLaMa-3
8B
10.3
21.5
11.4
27.6
17.7
IInstruct
70B
3.6
16.0
5.8
15.2
10.2
1.8B
33.5
37.9
24.7
13.3
27.4
4B
48.0
38.6
57.4
12.7
39.2
Qwen-1.5
7B
17.0
20.6
10.9
26.5
18.8
-Chat
32B
7.8
20.0
9.6
26.4
16.0
72B
10.9
22.6
9.6
24.7
17.0
110B
8.7
16.0
11.5
23.5
14.9
D FULL RESULTS OF REWARDBENCH
Table 5
14 15 present the full results of the reward bench. After inspecting the error cases, we categorize the performance drop in the Chat-Han Hard and Safety subsets into two main aspects:
16

---

Published as a conference paper at ICLR 2025
The cnrrucctionfollowinn capabilitiess become bit worse. For example, LLMs tend to solve the "user question" instead comparing two responses, or LLMs not output answers in requested formats, causing parsing failures when computing performance scores.
LMs overly focus helpfulness in safety prompts, therefore causing performance degrada- tion in the Safety dataset.
However, the positive effect of PINE (i.e., climinating position bias) is more significant than these negative effects; therefore, the overall PINE is still beneficial to models.
Table 5: Full results of Table Vanilla denotes the normal inference, (GT at A) means the ground truth chosen response is presented at the first, and (GT at B) indicates at the second. PINE consistently improves LM's performance across different model sizes. Consistent Table with severe position bias cyan. It can be observed that PINE generally improves performance on cyan subsets by large margin, which is consistent to our motivation and goal.
color the subsets
Model
Size
Method
Chat
Chat-Hard
Safety
Reasoning
Avg.
Vanilla (GT at A)
35.2
64.6
80.3
67.5
Vanilla (GT at B)
85.3
48.7
65.3
66.0
66.3
8B
Vanilla
85.3
41.6
67.0
65.3
64.8
LLaMa-3
PINE
85.6
41.5
66.5
73.4
+1.7
IInstruct
Vanilla (GT at
98.6
52.0
73.6
87.8
78.0
Vanilla (GT
93.9
62.1
69.8
80.3
76.5
70B
Vanilla
97.4
58.3
69.6
78.9
76.0
PINE
96.9
57.4
67.7
87.6
77.4 1.4
Vanilla (GT at A)
31.7
30.0
40.3
43.3
36.3
Vanilla (GT at B)
69.4
72.6
65.7
57.2
66.2
1.8B
Vanilla
49.7
50.9
52.0
48.4 4
50.3
PINE
30.0
59.9
61.4
60.1
52.9
Vanilla (GT at
24.8
17.4
42.8
29.5
Vanilla (GT at B)
86.6
74.5
82.9
62.3
76.6
4B
Vanilla PINE
58.9
48.7
50.9
54.1
53.1
73.0
45.2
53.7
61.0
58.2, 2..2..6.1
Vanilla (GT at A)
85.5
35.9
62.4
62.1
61.4
Vanilla (GT
77.1
47.4
59.5
541.3
59.6
7B
Vanilla
77.5
14.2
62.6
59.3
60.9
Qwen-1.5
PINE
85.8
38.7
58.6
63.0
61.5 0.6
-Chat
Vanilla (GT at A)
93.6
47.7
77.1
78.3
74.2
Vanilla (GT at B)
91.9
52.2
81.6
73.6
74.8
Vanilla PINE
92.7
51.2
80.5
66.8
72.8
93.0
49.8
79.7
76.7
74.8
Vanilla (GT
95.7
59.0
80.8
83.0
79.6
Vanilla (GT at
46.5
73.7
68.7
69.5
72B
Vanilla PINE
94.0
51.4
77.8
68.2
72.8
93.9
46.1
78.2
69.0
71.8-
Vanilla (GT at A)
97.5
71.9
85.7
93.7
87.2
Vanilla (GT at B)
95.0
67.5
83.4
76.0
80.5
72B (Qwen 2.5)
Vanilla PINE
96.6
68.0
83.3
85.5
83.4
96.6
67.1
83.0
91.3
74.5 1.1
Vanilla (GT at A)
70.5
90.0
Vanilla (GT at B)
91.1
59.2
79.5
73.0
75.7
110B
Vanilla PINE
96.2
66.7
83.7
78.0
81.1
64.88
85.0
86.2
82.9
17

---

Published as a conference paper at ICLR 2025
Table 6: Full version of Table PINE achieves superior performance baseline models, performing 4.8% and 4.7% better than the best performed baseline on two models,
Model
Method
Chat
Chat-Hard
Safety
Reasoning
Avg.
NIA (GT at A)
81.0
40.7
59.7
43.7
56.3
NIA (GT B)
81.0
49.7
65.8
66.7
65.8
LLaMa-3
NIA
80.9
46.7
64.0
55.9
61.9
8B-Instruct
PCW
78.6
46.8
64.8
56.5
61.7
SP
79.6
43.3
65.0
55.4
60.8
PINE
85.6
41.5
66.5
73.4
66.7
NIA (GT A)
67.7
57.2
59.6
60.7
61.3
NIA (GT at B)
67.9
35.9
61.0
44.1
52.2
Qwen-1.5
NIA
74.9
43.5
57.4
51.4
56.8
7B-Chat
PCW
67.2
42.0
58.3
53.4
55.2
SP
69.4
41.8
58.0
52.4
55.4
PINE
85.8
38.7
58.6
63.0
61.5 44.7
Table Meta-Lllama3--S--nnsrrttt results RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
91.0
90.0
alpacaeval-hard
91.6
90.0
alpacaeval-length
71.6
77.4
donotanswer
45.2
38.2
hep-opp
78.7
82.6
hep-go
77.1
86.6
hep-java
73.5
82.9
hep-is
74.4
84.1
hep-python
79.0
85.7
hep-rust
74.4
81.4
Immbar-adve-GGPTTn
23.4
24.5
Imbar-adder-GGPTO
63.8
67.0
llmbar-adee--aanee
40.2
34.8
aabbar-adver-neighbor
20.9
16.8
llmbar-natural
66.0
74.5
math-prm
54.5
62.8
mt-
mt-bench-easy
92.9
87.5
mt-bench-hard
68.9
64.9
mt-bench-med
83.8
80.0
refusals-dangenee
71.5
74.0
refusals-offenst
76.0
73.5
xstest-shoull-refusse
70.5
71.8
xstest-shoull-respond
72.0
76.4
18

---

Published as a conference paper at ICLR 2025
Table 8: 1aiimma---70-Insttttti Inssruct results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
100.0
100.0
alpacaeval-hard
100.0
100.0
alpacaeval-length
91.1
89.5
donotanswer
47.1
48.2
hep-opp
92.7
92.1
hep-go
89.9
97.0
hep-java
92.1
97.0
hep-js
93.3
95.1
hep-python
90.9
95.4
hep-rust
89.0
91.5
Ilmbar-adver-GPTTsss
55.4
57.6
Imbar-adver-GPPOuu
73.4
76.6
llmbar-adver-mmnuall
53.3
47.8
32.8
28.7
Ilmbar-natural
83.0
84.0
math-prm
66.4
80.5
mt-bench-easy
100.0
100.0
mt-bench-han
78.4
75.7
mt-bench-med
97.5
97.5
refusals-dangeron
63.5
62.5
refusals-offensive
66.5
66.5
sstest-should-refuss
68.8
63.3
xstest-shouuldrespoodd
96.8
96.4
Table 9: Qwen1.5-1..B- Bhttt results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
47.5
17.0
alpacaeval-hart
56.3
13.7
alpacaeval-length
50.0
45.8
donotanswer
54.0
52.6
hep-cpp
49.4
51.5
hep-go
54.3
48.8
hep-java
52.7
49.4
hep-is
48.2
47.6
hep-python
49.4
52.1
hep-rust
54.6
50.3
Mmbbaraddve-GGPThh
44.0
76.6
Imbar-adver-GPPT
55.3
40.4
Imbar-adver-maan manual
44.6
64.1
lmbar-adver-
56.7
66.8
Ilmarr-ntural
49.0
51.5
math-prm prmm
45.5
70.2
mt-bench-easy
39.3
57.1
mt-bench-hard
54.1
35.1
mt-bench-med
46.2
45.0
drfusals-dangeronnn
48.5
88.0
refusals-offensicc
49.5
54.5
rstest-should-refuse
53.2
53.9
xstest-shouuldrespood
52.2
68.8
19

---

Published as a conference paper at ICLR 2025
Table 10: Qwenl.5-4B-Chat results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
59.5
77.5
alpacaeval-hard
62.1
80.5
alpacaeval-length
60.5
70.0
donotanswer
54.4
18,4
hep-cpp
50.0
50.0
hep-go
50.3
51.8
hep-java
49.1
51.2
hep-js
49.7
49.4
hep-python
50.0
53.0
hep-rust
50.6
50.6
Ilmbar-adver- GPnnsst
36.4
38.6
Ilmbar-adver-GPTT GPrOut
54.3
51.1
Ilmbar-adver-mamull
51.1
39.1
llmbar-adver-neighhee
52.2
42.5
llmbar-natura math prm
48.0
53.5
58.2
70.9
mt- netbbench-easy
44.6
75.0
mt-bench-hard
58.1
48.6
mt- mt-bench-med
56.2
50.0
refusals-dangerous
43.0
31.0
refusals-offensive
47.0
71.0
xstest-should-refie
53.6
64.3
xstest-shouldrresponl
51.0
71.0
Table 11: Qwen 1.57-..t- results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
74.0
91.0
alpacaeval-hard
90.0
96.8
alpacaeval-length
65.8
74.7
donotanswer
19.9
11.0
hep-cpp
60.4
76.8
hep-go
61.9
69.2
hep-java
56.1
74.1
hep-is
59.5
68.6
hep-python
63.4
66.8
hep-rust
62.8
65.9
Ilmaar-avve- Bebbur--dver-GPPThmee GInnnst
40.2
23.4
Embar-addvr-GPPOO
53.2
45.7
Ilmbar-adver-mamull
35.9
35.9
lmmbar-adver-neighhoo
21.6
19.8
llmbar-natural
71.0
70.0
math-prm
57.9
55.8
mt-bench-easy
87.5
85.7
ut-bbnch-harrd
62.2
55.4
mt
nt--bench-me
77.5
72.5
refusals-
daageros
49.0
40.5
refusals-offensive
86.0
86.0
xstest-should-refisss
74.0
63.6
-stest-shoull-respond
75.6
86.6
20

---

Published as a conference paper at ICLR 2025
Table 12: Owenl.5-32B-Char results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
97.0
97.0
alpacaeval-hard
98.9
98.9
alpacaeval-length
81.1
82.1
donotanswer
44.5
41.2
hep-cpp
87.8
91.5
hep-go
80.5
93.9
hep-java
88.7
96.3
hep-js
84.5
95.7
hep-python
86.3
93.6
hep-rust
82.0
88.1
Ilmbar-adver- Seann--dver-GGPTiee GPTlnst
43.5
34.8
Ilmbar-adver-GPTT GPrOut
68.1
57.4
Ilmbar-adver-mamull
32.6
37.0
llmbar-adver-neighheee
25.0
28.4
llmbar-natura math prm
83.0
85.0
48.7
60.3
mt
netbbench-easy
96.4
92.9
mt-bench-hard
81.1
75.7
mt-
mt-bench-med
92.5
95.0
refusals-dangerous
80.0
80.0
refusals-offensive
99.0
99.0
xstest-should-redt
90.6
89.3
xstest-should-respon
84.4
85.6
Table 13: Qwenl. 1reenl.5-72B-har results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
98.0
98.0
alpacaeval-hard
97.4
97.9
alpacaeval-length
85.3
84.2
donotanswer
39.0
38.2
hep-opp
88.1
89.6
hep-go
85.4
92.1
hep-java
87.2
90.9
hep-js
90.9
90.9
hep-python
87.2
89.6
hep-rust
88.1
87.2
Imbar-adver--PPTIss
44.6
33.7
Imbar-aadver-GPPOOu
57.4
61.7
llmbar-adver-mmnual
41.3
39.1
lmmbar-adver-neighhor
28.0
20.9
Ilmbar-natural
84.0
81.0
math-prm
48.5
47.9
mt be-beenh-
96.4
100.0
mt-bench-haard
70.3
62.2
mt bebbbeennheee
95.0
92.5
refusals-dangerous
75.5
73.0
refusals-offensive
94.0
95.0
xstest-shoull-refius
87.7
91.2
xstest-shoulddrespond
86.8
85.0
21

---

Published as a conference paper at ICLR 2025
Table 14: Qwen2.5-72B-Insteee results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
99.0
99.0
alpacaeval-hard
97.9
98.9
alpacaeval-length
91.6
89.5
donotanswer
48.5
52.9
hep-opp
95.7
95.7
hep-go
97.0
98.8
hep-java
98.8
97.6
hep-js
94.5
98.2
hep-python
98.8
98.8
hep-rust
94.5
97.6
Ilmbar-adver-Oo GPTnss
66.3
68.5
Libaar-adver-GPTOu
76.6
72.3
lmmbar-adver-manual
65.2
63.0
Imbbar-adver-neighhorr
41.8
44.0
Iltburrnatural
92.0
87.0
math-prm
74.5
84.8
mt-
uu-bench-ease
100.0
100.0
mi-bench-hardd
94.6
91.9
mt --bbenhh-edd
97.5
100.0
refusals-dangerous
78.0
82.0
refusals-offensive
95.0
92.0
xstest-should
92.2
89.6
xstest-should-respe respond
95.2
93.6
Table Qwenl 1--1O0t results on RewardBench
Dataset
Vanilla
PINE
alpacaeval-easy
95.0
97.0
alpacaeval-hard
98.9
98.9
alpacaeval-length
93.7
88.4
donotanswer
51.5
55.9
hep-opp
87.8
92.1
hep-go
83.8
94.8
hep-java
86.6
94.8
hep-js
90.5
92.4
hep-python
83.8
93.9
hep-rust
85.7
90.9
Immar-adver-
70.1
65.2
Imbar-aadver-GPPOOu
72.3
61.7
llmbar-adver-mmnuall
60.9
65.2
lmmbar-adver-neighhor
44.8
41.4
Ilmbar-natural
86.5
90.0
math-prm
69.6
79.2
mt nt-bench-eassy
98.2
100.0
mt-bench-haard
83.8
83.8
mt
97.5
97.5
refusals-dangerous
76.0
84.0
refusals-offensive
97.0
97.0
xstest-should-refin
91.6
91.9
xstest-shoulddrespond
95.6
92.4
22

---

Published as a conference paper at ICLR 2025
E IMPLEMENTATION DETAILS
E.1 EXPERIMENT SETTING
For reproducibility. the generation temperature is set to O. We use PyTorch Ansel et al. 2024|] Paszke al. 2019) Transformers (Wolf et al. 2020 and vLLM (Kwon et al. 2023) for our experiments All experiments are launched with a single node of 8x A 100 80G with SXM connection. 70B and 1 10B models are launched with 3x and 4x A100, and other model sizes can be launched with 1x A100.
E.2 MOLECULE GENERATION AND MATH REASONING TASK DETAILS
Molecule Generation. In this task, the input contains several properties that are interchangeable, and LMs are asked to generate molecules that satisfy these properties. We train such an LM with QM9 (Ramakrishnan et al. 2014) dataset. The QM9 dataset collects over 130k 3D molecules with 3D structures (Li et al., 2024) calculated by density functional theory (DFT). Each molecule in QM9 has less than heavy atoms, and its chemical elements all belong H. C, N, O, F. We take six quantum property values as the conditional input to LMs and train LMs to generate molecules with the conditioned quantum property values. We split the training dataset of QM9 to two subsets where each subset has 50k samples, and train LMs and an EGNN-based quantum property prediction models (Satorras et al., 202T) on these two subsets, respectively. The six quantum properties are polarizability (ox), HOMO energy (<HоMO). LUMO energy (éLUMO), HOMO-LUMO gap (Ac), dipole moment (1) and heat capacity at 298.15K (C.).. The LM is 8-layer Llama model with attention heads and 768 hidden dimensions. To evaluate the performance, we sample 10000 sets of 6-property conditions, randomize the property order in each condition, and generate molecules conditioned on these property values by the trained LMM and compute the mean absolute difference (MAE) between the given property values and the property values the generated molecules. Note that we use the trained EGNN-based property prediction models calculate the property values of the generated molecules.
Math Reasoning. We R-GSM (Chen et al. 2024a) subset of GSM8K. This small dataset (which contains 220 problems) is designed to test LMs' performance with interchangeable premise orders. Problems in the dataset contain several conditions that do not have progressive relationship. Therefore, their positions are interchangeable. We further clean this dataset to remove problems where conditions do not read smoothly after changing positions (e-g., use pronouns in the first condition but introduce names in the second condition), yielding small set containing 95 problems. We test Qwen- 1.5 models on this dataset.
E.3 PROMPTS
We use the prompts provided by RewardBench [Lambert et al. 2024a) official repo for the LM-as-a- judge task:
System prompt:
impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not
Please act
"Customized license. https//giihbb.ccm/ppoorroyyyyoo "Apache-200 license. htw.::ggg......gooooooooeooooorm "Apache-200 license. http://gghhiaaccommm//mmmooooiii
23

---

Published as a conference paper at ICLR 2025
favor certain names of the assistants. Be objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better.
User prompt:
[User Question]
(The start of Assistant A's Answer]
[The End of Assistant A s Answer]
[The Start of Assistant B's Answer]
[The End of Assistant B's Answer]
We the prompts of official repo (Liu et 2024) for the retrieval- -uggmeted QA experiments:
User Prompt:
Write a high-quality one sentence answer the given question using only the provided search results (some of which might be irrelevant).
Document
(Title:
Document
(Title: ......)
Document (Title:
......)
Document (Title:
Question:
Molecule generation does not need prompts, We use prompts in OpenAA//imple--vall 10
to evaluate
R-GSM dataset.
F QUALITATIVE EXAMPLES
F.I LM-AS-A-JUDGE
[User Question]
Who created the Superman cartoon character?
[The Start of Assistant Answer]
The Superman cartoon character was created by Jerry Siegel and Joe Shuster in 1938.
[The End of Assistant A's Answer]
[The Start of Assistant 3's Answer]
Superman, the iconic comic book superhero, was created by writer Jerry Siegel and artist Joe Shuster. Superman first appeared in Action Comics #1, which was published by Detective Comics, Inc. (later Comics) in June 1938. The character's immense
24

---

Published as a conference paper at ICLR 2025
pepularity established him one of the most enduring and recognizable figures in the superhero genre.
[The End of Assistant B's Answer]
Here are outputs of Q
Both AI assistants provide accurate and informative responses to the user's s question. Assistant A's answer is brief and directly states the creators' names and the year of creation, while Assistant в offers additional context by mentioning the first appearance, the publisher, and the character's significance in the superhero genre. However, since the user asked for the creators, Assistant A's answer is more directly helpful in this specific context. Final verdict: [[A]]
Vanilla:
[[B]]. Assistant B's response is more detailed and informative, providing additional context about the origin the character in Action Comics #1, the publisher, and its
Vanilla (Swapped):
significance ficance in the superhero genre. It also includes the names the creators, which is crucial information for the question. Assistant A's response is shorter and lacks the historical context and specific details. Therefore, Assistant B is better in answering the user's question.
[[B]]. Assistant В provides a more comprehensive and detailed response to the user's question. It not only mentions the creators but also provides additional information about
PINE:
the character's debut and the company that published the comic. This answer is not only accurate but also offers context. that the creation. Assistant A's
highlights the significance response is brief and lacks the depth and historical context provided by Assistant
F.2 RETREEVAL-AUGMENTED QUESTION-ANSWERIN
User Prompt:
Document (Title: Pearl) south Pacific and Indian Ocean. The largest pearl oyster is the "Pinctada maxima", which is roughly the size of dinner plate. South Sea pearls are characterized by their large size and warm luster. Sizes up to 14 mm in diameter are not uncommon. In 2013, Indonesia Pearl supplied 43 percent of South Sea Pearls international market. The other significant producers are Australia, Philippines Myanmar and Malaysia. In 1914, pearl farmers began growing cultured freshwater pearls using the pearl mussels native to Lake Biwa. This lake, the largest and most ancient in Japan, lies near the city of Kyoto. The
Document (Title: Laccadive Sea) the gulf as most productive in the world. Although extraction of natural pearls is considered too expensive most parts the world, in the gulf. Also collected in large numbers are Shankha mellusks ("Xancus pyrum") whose shells are used as a ritual and religious object. Other mollusks of the sea are either too scarce not popular in the Indian society and therefore have no commercial value. Another traditional occupation in the Laccadive Sea is fishing. The annual fish eatch is 2, 000 to 5, 000 tonnes from the Lakshadweep islands, which is mostly constituted by tuna
is still conducted
Document (Title: Pearl) including the Cook Islands and Fiji are being extensively used for producing cultured pearls. The rarity
25

---

Published as a conference paper at ICLR 2025
of the black cultured pearl is now a "comparative" issue. The black cultured pearl is rare when compared to Chinese freshwater cultured pearls, and Japanese and Chinese akoya cultured pearls, and is more valuable than these pearls. However, it is more abundant than the South Sea pearl, which is more valuable than the black cultured pearl. This is simply because the black pearl oyster "Pinctada margaritifera" is far more abundant than the elusive, rare, and larger south sea pearl oyster "Pinctada maxima", which cannot
Document (Title: Pearl powder) Pearl powder Pearl powder () is a preparation of crushed pearls used in China and elsewhere for skin care and in traditional Chinese medicine. Pearl powder is made from freshwater pearls or seltwater pearls below jewellery grade. These are sterilised in boiling water and then milled into fine powder using stainless steel grinding discs or by milling with small porcelain balls in moist conditions. The powder is sold as such or mixed into creams. Pearl powder is widely believed to help improve the appearance of the skin, and is used as cosmetic by royal families in Asia. It
Document (Title: Hyderabad pearl) with white pearls. Recently, several pearl makers are exporting processed pearls to markets Europe and the US. With the capital that they gain from this marketing, they are able to purchase machinery for advanced refinement. In particular, equipment that uses enzymes present in thermophiles is able to substantially improve the process of refining pearls. Hyderabad pearl Hyderabad
consideres the main pearl trading center in India. The most notable area devoted to the trade is the village called Chandanpet just outside Hyderabad, wherein almost the entire population is engaged in the delicate art of drilling pearls, a skill they
Document (Title: Pearl) pearls". The correct definition of South Sea pearl { as described by CIBJO and GIA { is a pearl produced by the "Pinctada maxima" pearl oyster. South Sea pearls are the color of their host "Pinctada maxima" oyster { and can be white, silver, pink, gold, cream, and any combination of these basic colors, including overtones of the various colors of the rainbow displayed in the pearl nacre of the oyster shell itself. South Sea pearls are the largest and rarest of the cultured pearls { making them the most valuable. Prized for their exquisitely beautiful órientór lustre,
Document (Title: Chandrani Pearls) year 2007(08 Chandrani Pearls imported their pearls from Japan, China or Korea. Chandrani Pearls Chandrani Pearls is prominent pearl jewelery brand of India. It pioneered the concept of pearls in India. Chandrani Pearls's headquarters is at Kolkata in West Bengal. Chandrani Pearls was started on 24 January 1985 by Mr. Kuldip Nayar, his wife Mrs. Lakshmi Nayar and his father late Mr. N.C. Nayar in Kolkata's up market Minto Park area. Chandrani Pearls management is now assisted by Nisheeth Nayar, sons of Mr. Kuldip Nayar. Chandrani Pearls have 63 showrooms across states. From modest turnover of Rs.
Document (Title: Pearl) For thousands of years, seawater pearls were retrieved by divers in the Indian Ocean in areas such as the Persian Gulf, the Red Sea and the Gulf of Mannar. Evidence also suggest a prehistoric origin to pearl diving in these regions. Starting in the Han Dynasty (206 BC(220 AD), the Chinese hunted
26

---

Published as a conference paper at ICLR 2025
extensively for seawater pearls in the South China Sea. In the 14th-century Arabian Sea, the traveller Ibn Battuta provided the earliest known description of pearl diving by means of attaching a cord to the diver's 8 waist.
Document (Title: Chandrani Pearls) Chandrani Pearls Chandrani Pearls is a prominent pearl jewelery brand of India. It pioneered the concept of pearls in India. Chandrani Pearls's headquarters is at Kolkata in West Bengal. Chandrani Pearls was started on January 1985 by Mr. Kuldip Nayar, his wife Mrs. Lakshmi market
Nayar and his father late Mr. N.C. Nayar in Kolkata's Minto Park area. Chandrani Pearls management is now assisted by Nisheeth Nayar, sons of Mr. Kuldip Nayar. Chandrani Pearls have showrooms across 9 states. From a modest turnover Rs. 2.75 lakhs in 1990 to turnover of Rs. 15.00 crores in
Document (Title: Hyderabad pearl) Hyderabad pearl Hyderabad is considered the main pearl trading center in India. The most notable area devoted to the trade is the village called Chandanpet just outside Hyderabad, wherein almost the entire population is engaged in the delicate art of drilling pearls, a skill they have practiced for generations. This practice also makes the city one of the largest pearl drilling locations in India. The pearl industry in Hyderabad flourished due to the patronage of the Qutub Shahi kings and the Asaf Jahis, who were said to have an affinity for sparkling jewels. The pearls were not only part of
Question: in which sea pearl is found in india?
Here Llama-3-70B-Insruutt results:
Vanilla: According to the search results, pearls are found in the Laccadive Sea India.
PINE: Pearls are found in the Indian Ocean, specifically in the
Gulf of Mannar and the Laccadive Sea, off the coast of India.
F.3 MOLECULE GENERATION
The task has properties as input, and molecule structure as output. Below is one example:
Input:
alpha*81. 68* *8 mu*2.
0332* homo=-55 -..xxx5.87453 fapx7.8668.
lumo 1. 9919*
Output: 0.00 0.00 0.00 0 0.96 1.57 0.00 C 1.95 2.33% 0.00
2.35 2.50% 7. H 2.32 2.53 -0.71 3.22 2.00 -0.02%
3.46
1.80a 0.249
4.50 2.23* 0.00 H 4.99 2.22 0.26 и 4.55 2.46
-0.12a с 5.58
2.08 -0.18 H
2.07
-0.09- N 5.89 2.05e 46.46
5.57 2.18
-0.710
H 4.57 2.25
-0.78-
н 6.29 2.32 -0.72 5.98
2.06
-0.86*
C 5.26 1.85
-0.34
H 6.06 1.72 -0.33 c 3.81 1.74
-0.36
H 3.94
1.47 -0.99 3.58
1.75
-0.55
F.4 R-GSM
R-GSM is just subset of GSM8K, with the premise order changes. Here is an example input:
Input: Carmen goes to auction to win antique desk. The bids on the desk rise by 50 each time and 3 other people each bid once. She accepts the opening bid of 200 and continues bidding until she wins. Carmen bids after each of the 3 other people and eventually wins. How much money, in dollars, does the desk cost her?
27

---

Published as a conference paper at ICLR 2025
Here The bids on the desk rise by 50 each time and 3 other people each bid once. and She accepts the opening bid of 200 and continues bidding until she wins. are interchangeable.
28


Reviews:
[{"review_id": "l4cOSOy8Jy", "reviewer": "Reviewer_azrE", "summary": "The authors introduce a method to de-bias a positional dependence for concatenated documents in the context of a LM and prove its supremacy against other de-biasing baselines.\n\nThey also compare against vanilla inference, but I cannot tell whether the numbers are actually beneficial.", "questions": "- Please discuss the relationship to NoPE (https://arxiv.org/pdf/2305.19466) in your related work\n- I find the importance measure that you introduce a little bit weird if you make it independent of positional bias, since you will later calculate attention with positional bias in there, which might change the importance quite a bit. Have you found your measure of document importance, without positional bias, to be monotonic with an importance measure that calculates the attention with positional bias, but putting each document candidate at the same position, right before the currently decoded document for instance?\n- I am overall confused by the introduction of bidirectional attention. Given that you calculate importances independently of relative document-position, and you reorder anyway, why do you have to have an attention in the forward direction, instead of simply always putting the document you are decoding at the last position? I might have understood this wrong, the first half of Page 5 is a little confusing to me. When following the concrete example in the proof, there is no bidirectional attention, right?\n- Can you comment on the downsides of making keys query dependent (L235)? This seems like you will loose parallelism, at the very least the option to KV-cache.\n- Can you comment on your intuition as to why SP and PCW are worse than your approach?\n- You say you introduce two other debiasing baselines, permutation and calibration. Is permutation the same as Vanilla (shuffle) in Figure 4? Why are the numbers for it not in any tables?\n- Table 2: I don’t fully understand the baselines. IIUC, the GT is either at A or at B, which is why you say 50% random guess baseline, i.e. there is no position C. Why is Vanilla (without qualifier) worse than both Vanilla (GT at A) and (GT at B)? it is not always one of the two, which would lead to some kind of average?\n- what do you mean by reasoning “pairs” in 418ff?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- very relevant problem\n- the recency and primacy biases are interesting.", "weaknesses": "- Figure 2 is confusing, since the documents change color when you move them (in the right plot). So it looks like the $D_3$ in the first row is the tokens [2,3], but in the second assignment iteration suddenly $D_3$ is [4,5]? It seems inconsistent with the last row, where importance ordering is $D_2 > D_3 > D_1$, but they are not ordered in that way.\n- I don’t think you can claim that you pinpointed the causes of positional bias in transformers. Everybody knows what those are.\n- Please Correct me if I am wrong here.\nThe proof in App B shows that the softmax term is invariant under inter-document permutation before the ordering takes place, but Lemma 1 takes the QKV invariance as a given.\nHow I think Lemma 1 and the proof should go:\nLemma 1: “The PINE algorithm makes $H_{Pine}$  document-position invariant”\nProof: “The algorithm sorts the documents by a document-position invariant metric and concatenates them. Therefore, the resulting concatenation  is invariant. $H_{Pine}$ is only a function of that concatenation, therefore $H_{Pine}$ is invariant. Proof ends”.\n- I can’t tell if the numbers are good. The method outperforms other methods of debiasing, but it does not seem to do super well against Vanilla (shuffle) in Figure 4 for instance. Given that you shuffle reward bench, is Vanilla in Table 2 also implicitly Vanilla (shuffle)? It is overall confusing to me, but I admit that this is an opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CnClJSl4Y2", "reviewer": "Reviewer_K8Th", "summary": "Positional bias is a major problem for language models. Several proposals have been made in literature towards mitigating this issue. This paper proposes one method by making the inter-document attention bidirectional. Specifically, they propose their own order such that the effects of order dependence in positional encoding is minimized.", "questions": "- Figure 2, I understood the last row. 8 is at its original place and (4,5)> (6,7) > (2,3). \n    - Rows 6 and 7 -- (6,7) is at its original place and (2,3) > (4,5)\n    - Rows 4 and 5 -- (4,5) should be at their original place, but (6,7) is! Why revise the position of this document to the end when both the documents can be (equally) at small distance from the document under consideration?\n- I only understood Figure 2 after going through paragraph 2 in lemma 1. Please consider revising the write up with the document order for computation of each of the rows of the attention.\n\n### Suggestions:\n- Move paragraph 2 from Lemma 1 proof from appendix to main paper to explain your method.\n- Consider revising the words position invariant to input-position invariant. Please also consider revising the claim that the position bias of LLMs is eliminated (in the title of the paper). From my best understanding, it is mitigated through a reordering and not eliminated.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- I like this paper. they propose a bidirectional attention in the input documents to achieve input order independent attention. \n- Their results are solid. For order dependent benchmarks, they do achieve significant gains.\n- While the paper is solid, I would like the authors to revise their claims of position invariance to input-position invariance.", "weaknesses": "- I tried understanding Figure 2 many times, it was not clear to me. Please see question below. The only sensible explanation is that the attention is simulating the document under question being placed at the end of the sequence, with nearness being provided by importance scores.\n- What about the n^2 importance score computations per set of n documents. How to do this when there are multiple documents?\n- The position invariance is quite misleading. The correct term I believe would be importance dependent position weighting. i.e. You know the documents far away to the current document under consideration have higher likelihood of error, and hence you place them far away, such that the impact of position is mitigated. \n- Here is a counterexample for position invariance: Let us take a query which involves multi-hop retrieval. The query asks: Retrieve the home pages of all the professors working in the CS department of X university, and list out their topics of interest. Here the retrievals may be all nearly equally important. A full position invariance would require an independence with order whereas that cannot be achieved through this method. What you instead do is to rank documents by importance order and then mitigate positional dependence by placing them in that order of closeness for attention. This is not \"invariance\". My major concern is with the nomenclature, which claims invariance, and eliminating position bias.\n- Lemma 1 may need a revision: sorting by computed importance scores introduces an implicit positional bias. While the function \\( f(\\text{input order}, \\text{content}) \\rightarrow \\text{output permutation order} \\) maintains input order invariance if \\( f(\\text{io1}, \\text{content}) = f(\\text{io2}, \\text{content}) \\) for all input orders \\(\\text{io1}, \\text{io2}\\), it imposes a new order-dependent bias in the output, meaning true position invariance is not achieved. Thus, while input-position invariance is maintained, true positional invariance is not achieved, as the sorting process introduces a new, order-dependent bias in the output.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "69Dy9fGixL", "reviewer": "Reviewer_Qpxn", "summary": "The paper presents a modification to the causal attention mechanism of decoder-based transformers. The proposed mechanism adds bidirectional attention to each document, in order to make representations position-invariant. New bidirectional attention terms are added in blocks (for the tokens corresponding to each document), which are ordered according to their importance to the document on the diagonal. The proposed importance score is based on a version of the standard attention calculation that ignores positional differences. The paper proves positional invariance and evaluates the empirical performance of the new method on a set of tasks.", "questions": "1. Your method gives position invariance as long as the attention orderings for each document depend only on the content of the documents rather than their positions. Thus, mathematically you can consider any function that maps a list of k documents to a permutation on k-1 documents (k-1 since the document on the diagonal is always in the last position). This class of functions is large and includes special cases like variants of your importance score (e.g., variants regarding the choice of normalization). It seems important to assess the sensitivity of PINE with respect to the choice of ordering function.\n\n2. If position bias is as big a problem for our field as the paper argues, it should be possible to show gains on mainstream benchmarks. Consider the generic position-dependent biases in few-shot prompting, as identified by Zhao et al. in their contextual calibration paper. Could you improve, say, the recency bias of few-shot prompting with your approach? If yes, it would be nice to see this demonstrated on a mainstream LLM benchmark.\n\n3. I am not very familiar with the Qwen series of models. Since your method yields essentially no gains for Llama on RewardBench (as noted above – please correct me if my analysis is mistaken), I wonder if the strong performance on the Qwen models has something to do with the architecture of these models. After looking into the matter, I wonder if the Qwen models’ use of sliding window attention could be the reason why the extra bi-directional attention from your method yields a benefit. What do you think? \n\n4. The ability to permute documents gives you a way to estimate the variance of vanilla sampling performance. Given the wide disparity between the GT-A and GT-B performances on RewardBench, did you look into randomizing the order of premises on R-GSM? This seems especially important because there are so few samples (only 95) in this data set.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and the graphics are appealing.\n2. Position-invariant representations are nice, and you obtain them without k! overhead.\n3. The positional bias on RewardBench appears significant and adds credibility to your argument that position bias is an important problem.\n4. Strong results for the Qwen models on RewardBench.\n5. Strong results for Llama on the reasoning subset of RewardBench.", "weaknesses": "1. You compare your method to other mechanistic methods with different goals: for example, PCW helps extend the context window; NIA gives a speed boost by making the attention computation sub-quadratic. Thus, these are both *approximate* methods. Your method, by contrast, introduces extra overhead compared to standard sampling. For a fair comparison, it is important to look for methods that are more resource-intensive than standard sampling (for example, if your method has 8x overhead, a simple baseline could involve sampling 8 permutations of the documents and running vanilla sampling 8 times, followed by some aggregation scheme like majority voting).\n\n2. I appreciate the analysis shown in Figure 4b) but I would also like to see an analysis of the choice of importance score (see question 1. below).\n\n3. On both RAG-QA and RewardBench, PINE does not seem to improve over vanilla sampling with the Llama models. To see this, note that the average of the GT-A and GT-B scores should give the expected performance for vanilla sampling (with randomized ordering of the two documents -- please correct me if I am wrong). On molecular generation and math reasoning, you switch between evaluating the Llama and Qwen models, which does not inspire confidence; I would like to see the same list of models evaluated for each task. Overall, your selection of tasks seems somewhat contrived. See question 2. below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fynSBxJyp7", "reviewer": "Reviewer_1Z1E", "summary": "Proposed work introduces PINE, an attention mechanism that computes the attention scores in a positionally invariant manner. The method does not require any training and can be plugged into an transformer inference setup with minimal costs. Improved results in multiple benchmarks show the effectiveness of the approach.", "questions": "<none>", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The approach does not require any extra training, and can be used by any attention mechanism.\n- Remarkable improvements on rewardbench evals.\n- Comparison with relevant baselines and experiments with multiple models performed.\n- Positional invariancy is thoroughly ascertained through empirical and theoretical explanations.", "weaknesses": "- Regarding,\n> Hsieh et al. (2024) assumes that the position bias and real relevance are linear combinations and propose solutions accordingly. Different from them, we aim to eliminate the position bias from the mechanical perspective without any assumption at a reasonable cost.\n\nAlthough it is claimed that this solution is better motivated than the cited one, a comparison is still required since both papers are solving the same problem and show very good improvements in similar yet different tasks.\n\n\n- [This](https://openreview.net/pdf?id=gEMLMMG0m9) work debiases positional bias in the attention matrix by averaging attention values of documents at different positions. A mention of why such simple methods to debias are less relevant or a comparison is required.\n\n\n- Regarding line 288,\n> the extra big O computation complexity to obtain hidden states is O(nk log k),\n\nFrom my understanding, for an end-to-end inference time overhead calculation, this should be multiplied by #layers in the model. And further with #heads if computation across heads is not parallelized. The equation does not capture this.\nHence for lines 518-519,\n> we find the wall time of PINE is ~2x and ~8x of the vanilla inference\n\nThese numbers should depend on the size of the models being considered. Can you tabulate this for each model separately to give a better picture of overhead time? This is significant as benchmarks tend to use large models as evaluators and an inference time $\\propto $ #layers limits the practical applicability of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8cGPvXfhX", "reviewer": "Reviewer_1nRN", "summary": "This work addresses the position bias problem in language models (LMs), where models prioritize information based on its order in the input, affecting performance and reliability across applications. Through a mechanistic analysis, the study identifies causal attention and relative positional encodings as primary contributors to this bias. To mitigate it, the authors introduce Position-INvariant InferencE (PINE), a zero-shot approach that replaces causal attention with bidirectional attention between documents and uses attention values to determine document order. PINE effectively enhances model performance in tasks like QA, molecule generation, and math reasoning, showing notable gains in reasoning benchmarks where it surpasses even state-of-the-art models like GPT-4 in specific evaluations.", "questions": "1. Do different position assigning strategies have a big effect on the downstream performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a simple yet effective and novel method to mitigate the postion bias problem in LLMs.\n\n2. Extensive experiemnts on Llama-3 and Qwen demostrate the effectiveness of the proposed method.", "weaknesses": "1. The ablation study of the strategy of re-assign positions is required to better discuss the potential of further improving the method.\n\n2. There is no baseline comparison with other calibration based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)