Eliminating Position Bias of Language Models: A Mechanistic Approach

## Abstract

Abstract Position bias has proven to be a prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given context. This bias often leads to unexpected model failures and hurts performance, robustness, and reliability across various applications. Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and position embedding.
Based on the analyses, we propose to eliminate position bias (e.g., different retrieved documents’ orders in QA affect performance) with a training-free zero-shot approach. Our method changes the causal attention to bidirectional attention between documents and utilizes model attention values to decide the relative orders of documents instead of using the order provided in input prompts, therefore enabling P osition- IN variant inferenc E ( PINE ) at the document level.
By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning. Notably, PINE is especially useful when adapting LMs for evaluating reasoning pairs: it consistently provides 8 8 8 8 to 10 10 10 10 percentage points performance gains, making Llama-3-70B-Instruct perform even better than GPT-4-0125-preview and GPT-4o-2024-08-06 on the RewardBench reasoning set. 1 1 1 Code available at: https://github.com/wzq016/PINE . 1 University of Illinois Urbana-Champaign, 2 Harvard University, 3 Texas A&M University. Contact: ziqiw9@illinois.edu

## 1 Introduction

Language models (LMs) [, , , ] demonstrate impressive performance in general language tasks such as dialogue [], reasoning [], and schema induction [].
However, they tend to favor content at certain positions [, , , , , , ], which harms complex reasoning [], long-context understanding [] and model-based evaluation []. For example, LMs tend to favor the first when it is required to compare the quality of two candidate responses [], which hurts their reliability when being used as evaluators (Figure 1 upper); vision-language models perform better in the recognition when the target content is presented at the bottom of the image (Figure 1 lower right, see more examples in Appendix A). Different from ad hoc solutions from previous works [, , , , ], we seek to understand the causes of position bias and propose to eliminate (not just mitigate) the position bias without training and searching.

We start by analyzing the key components of state-of-the-art LMs – Casual Attention and Position Embedding. They are the key to the success of Transformers [], and are also the only two operations in Transformers [] that will bring undesirable position bias.This claim could be obtained through a concise logical proof (Section 3.2). Moreover, we find an interesting phenomenon through simple experiments: the most popular Rotary Position Embedding [] is shown to have recency bias [, ] due to its long-form attention weight decay w.r.t. the increase of relative positions, and the causal attention forces unidirectional information propagation, enabling models to pay more attention to distant content. Figure 1 lower left shows this retrieval-augmented QA [] experiment. The height change of the yellow area and orange area reflects the position bias of causal attention and RoPE. Since the yellow area is mostly wider at the beginning and the orange area generally becomes wider at the end (except for the last data point), this shows that the causal attention generally tends to favor distant content, while RoPE generally tends to favor nearby content.(^2^22More supporting experiments to this hypothesis in Section 4.3.)

Since the attention and position embedding are the causes of position bias, we propose PINE that can eliminate position bias by manipulating causal attention and RoPE to attend to different content equally. Take the retrieval augmented QA [], a task requiring LMs to answer questions based on retrieved documents, for example. The orders of retrieved documents should not affect the final results. To achieve this, we make the inter-document attention bidirectional so that the attention mask will equally attend to all documents.
Next, we compute importance scores between documents and use them to re-assign document positions so that positions in the original inputs are discarded.
We prove resulting approach enables Position-invariant inference (PINE) w.r.t. documents in a training-free/zero-shot manner.

To justify the effectiveness of PINE, we select four tasks: LM-as-a-judge [], which prompts LMs to choose the more helpful one from two given responses to a question; retrieval-augmented question-answering []; molecule generation based on provided properties; and math reasoning. In different tasks, “documents" have different meanings: responses in LM-as-a-judge, properties in molecule generation, and conditions in math reasoning. Notably, we find our method especially useful when LMs are used to assess reasoning pairs: PINE with Llama-3-70B-Instruct perform even better than GPT-4-0125-preview and GPT-4o-2024-08-06 on the RewardBench [] reasoning set.

To summarize, we:

- •
We pinpoint the causes of position bias in transformers: causal attention and position encoding (Section 3.2). We then propose a training-free approach dubbed PINE that can eliminate (with proof) the position bias given documents presumed to be position-invariant (Section 3.3).
- •
Four popular tasks across the general domain to expert domains (chemistry and math) show PINE can bring performance gains consistently across different models and sizes.

## 2 Related Work

Position Bias in LMs.
Position bias widely exists in LMs
[, , , , , , ]. The LM-as-a-judge task offers models two candidate responses to a question and asks models to select the more helpful one. It turns out that LM has a primacy bias that tends to favor the first response [].
Retrieval-augmented QA asks LM to answer a question based on retrieved documents. Liu et al., [], Peysakhovich and Lerer, [] find that LMs are prone to answer correctly when the document that contains the correct answer is presented at the beginning and the end of retrieved documents. Zheng et al., 2024a [] points out that models favor options at certain positions (e.g., prefer “A”) in multiple-choice QA. In the in-context learning task, Zhang et al., 2024a [], Xu et al., [] find that the order of in-context examples affects the final performance. Recently, several papers have proposed to understand the nature of position bias through prompting [] and calibration []. Our paper analyzes the phenomenon from the mechanical perspective: the computation must be positional-invariant to eliminate position bias.

Position Bias Solutions in LMs.
Mitigating position bias have been studied by many literature from many aspects, such as data augmentation with training [, ], content sorting by attention value during inference [], searching [, ], and calibration []. Moving one step forward, some other solutions are designed to eliminate position bias. Zheng et al., 2024a [], Zheng et al., 2024b [] use permutation then average on classification tasks, which will have unacceptable $\mathcal{O}(k!)$ ($k$ is the number of segments) computational overhead when $k$ is large. Hsieh et al., [] assumes that the position bias and real relevance are linear combinations and propose solutions accordingly. Different from them, we aim to eliminate the position bias from the mechanical perspective without any assumption at a reasonable cost. Although several existing approaches are from the mechanical perspective [, , ], they only perform well in classification tasks and fail in a more general setting: language generation.

## 3 Methodology

We start by running an example to illustrate position bias, followed by analyzing the cause of position bias, and end with our own approach PINE.

### 3.1 Formulation

We take retrieval-augmented QA as an example, where current LMs’ performance may greatly suffer from position bias []. The task requires the model to answer a question based on a set of given retrieved documents, where only one of them contains the correct answer. The system prompt SYS is: “Write a high-quality one-sentence answer for the given question using only the provided search results (some of which might be irrelevant).”. Given a question Q, and three retrieved documents: $\mathcal{D}_{1}$, $\mathcal{D}_{2}$, and $\mathcal{D}_{3}$, we can formulate several different inputs.
For example, $[\textsc{SYS}|\textsc{Q}|\mathcal{D}_{1}|\mathcal{D}_{2}|\mathcal{D}_{3}]$, and $[\textsc{SYS}|\textsc{Q}|\mathcal{D}_{2}|\mathcal{D}_{3}|\mathcal{D}_{1}]$.
We expect models to have the same output for these inputs because $\mathcal{D}_{2},\mathcal{D}_{3},\mathcal{D}_{1}$ are position-agnostic documents: their relative order is not supposed to affect the final result. However, the current LMs answer differently when presented with these different inputs and tend to answer correctly when the document contains the answer at the beginning or at the end of all documents [].
The systematic differences of model outputs caused by relative positions of documents reflect the position bias of the model. Therefore, current LMs cannot conduct inter-document position-invariant inference, and our goal is to make the inference invariant w.r.t. relative document orders. In the rest of this section, we will use this running example. However, we emphasize that “documents" have different meanings in different tasks: responses in LM-as-a-judge, properties in molecule generation, and conditions in math reasoning. Therefore, readers should be aware that the method is not just designed for a single task. In the rest of the paper, we merge SYS and Q into SYS for simplicity.

### 3.2 Causal Attention and Position Embedding Are The Cause of Position Bias

Feed-forward networks (FFNs), Query, Key, and Value (QKV) projections, and layer normalization in the Transformer architecture do not cause position bias, as they yield the same representations regardless of document positions.
Rather, the attention computation that leads to the position bias:

$$ $\displaystyle\mathbf{Q}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,.5,0}PE}}}=\text{{\color[rgb]{1,.5,0}\definecolor[named ]{pgfstrokecolor}{rgb}{1,.5,0}PE}}(\mathbf{Q},\mathbf{pos}_{\mathbf{Q}}), \mathbf{K}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb} {1,.5,0}PE}}}=\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{ rgb}{1,.5,0}PE}}(\mathbf{K},\mathbf{pos}_{\mathbf{K}})$ (1) $\displaystyle\mathbf{H}=\text{Softmax}\left(\mathbf{Q}_{\text{{\color[rgb]{ 1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}\mathbf{K}_{\text{ {\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}^{T}/ \sqrt{d}\right)\odot{\color[rgb]{0.21,0.49,0.74}\definecolor[named]{ pgfstrokecolor}{rgb}{0.21,0.49,0.74}\mathbbm{1}_{\text{causal}}}\mathbf{V}$ $$

where $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{n\times d}$ are queries, keys, and values, PE donotes the position encoding, $\mathbf{pos}_{\mathbf{Q}}$ and $\mathbf{pos}_{\mathbf{K}}$ denote the position of queries and keys, and $\mathbbm{1}_{\text{causal}}$ denotes the causal attention mask. Eq. 1 reveals that (1) the PE function yields different representations for documents if their orders changes, therefore affecting the importance score $Q_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}
PE}}}K_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{
1,.5,0}PE}}}^{T}$ and hidden states; (2) the $\mathbbm{1}_{\text{causal}}$ generates different attention masks for the input documents if we change their positions, resulting in different hidden states. To achieve inter-document position-invariant inference, $\mathbf{H}$ must remain the same regardless of documents’ orders.

### 3.3 PINE: Inter-Document Position-Invariant Inference via Bidirectional Attention

Figure: Figure 2: PINE: inter-document position-invariant inference via bidirectional attention. The attention matrix of the running example in Section 3.1 is at the left of the figure, the orange, different blue, and green colors denote system prompts (1 token), three different documents (2 tokens each) and decoded tokens (1 token), respectively. The number at $(i,j)$ in the figure, $p_{ij}$, denotes the position of a token $j$ when computing the attention from query $\mathbf{q}_{i}$. Therefore, $p_{\cdot j}$ is equal for all $i$ in vanilla inference. PINE enables inter-document bidirectional attention and then uses attention scores between documents to compute their importance. Then, documents are sorted by importances: more important documents are placed in closer positions. The computation of “importance score” is introduced in Section 3.3.
Refer to caption: x2.png

Our goal is to obtain an inter-document position-invariant hidden state $\mathbf{H}_{\text{PINE}}$, which does not change regardless of document orders. We can mechanistically eliminate the position bias by equally attending to all documents. Therefore, we propose PINE, an approach that uses bidirectional inter-segment attention and re-assigning positions by importance scores (computed from attention score) to eliminate position bias (Figure 2).

Bidirectional Attention. We first change the attention mask so that documents can attend to each other. Specifically, we make the inter-document attention bidirectional
but keep the intra-document attention causal (Figure 2, middle). Our goal is to eliminate “inter” position bias among different documents rather than “intra” position bias within each document. The latter will lose the order information of tokens, and models can degenerate into bag-of-words models, which is not what we expect.

Re-assign Positions: Sorting By Importance Scores. Re-assigning positions must consider two folds: the position of queries and keys. Each token in conventional LMs has the same position when serving as both query and key. In the bidirectional attention we use, this assignment has to be reconsidered. First, LMs are trained causally, meaning the position of the query must be larger than the keys in the attention computation. Therefore, it is necessary to manipulate positions so that each document is the last document when serving as queries (the diagonal of the rightmost figure in Figure 2). For tokens before and after documents, their positions are not affected when serving as queries.

Re-assigning positions for keys must be redesigned to eliminate position bias. We determine the positions of documents based on importance scores when they serve as keys (numbers in the rightmost part of Figure 2). Specifically, we first compute the attentions without position embedding involved: $\text{Importance}_{\text{token}}(i,j)=\text{Softmax}(\mathbf{q}_{i}\mathbf{k}_
{j}^{T}/\sqrt{d})$, where $d$ is the hidden state dimension. Then, we obtain the importance score between documents by aggregation. For example,
$\text{Importance}(\mathcal{D}_{1},\mathcal{D}_{2})=\sum_{i\in\mathcal{D}_{1},j
\in\mathcal{D}_{2}}\text{Importance}_{\text{token}}(i,j)/|\mathcal{D}_{2}|$. The length normalization is to prevent assigning higher importance scores to longer documents. The importance score could also be computed between individual tokens (e.g., Token 8) and documents. Lastly, we re-assign positions by importance scores as shown in the rightmost part of Figure 2: more important documents will have closer positions to the query. The rightmost part of Figure 2 shows the concrete position re-assignment for keys (its diagonal also represents the position re-assignment for queries). To avoid confusion, we address the fact that we do not actually sort tokens and only re-assign them to different positions. In our position re-assignment, the position of keys may vary depending on the queries (numbers in column are different), which is the key difference between PINE and vanilla inference. Besides, our method is not limited to specific position embedding types.

Inter-Document Position Invariant Inference. Once we have new attention mask and position re-assignment, we can place them into Equation 1, and obtain $\mathbf{H}_{\text{PINE}}$. By applying $\mathbf{H}_{\text{PINE}}$ to every layer, attention heads, and tokens, we reach our method PINE. We prove that:

###### Lemma 1 .

If the input $\mathbf{Q},\mathbf{K},\mathbf{V}$ are inter-document position-invariant representations, then $\mathbf{H}_{\text{PINE}}$ are also inter-document position-invariant representations.

###### Theorem 1 .

Given an input, if $\mathbf{H}_{\text{PINE}}$ is applied to every layer, attention head, and token to replace the conventional attention computation, then the model outputs are inter-document position-invariant representations.

The lemma can be proved by showing that (1) importance scores and position re-assignment are not a function of input document positions, and (2) bidirectional attention mask is not a function of document positions. The theorem can be proved by mathematical induction by (1) lemma, (2) FFN, QKV projection, and layer norm yield representations that are not a function of document positions, and (3) the embedding representation is not a function of document positions. We put the complete proof in Appendix B. We also suggest you read the proof if you feel unclear about the workflow of PINE , as we will go through a concrete example in the proof.

Some takeaways that are worth noting: (1) Both bidirectional attention mask and position re-assignment are needed to complete the proof. (2) PINE needs to be applied to every layer, attention heads, and tokens to complete the proof. (3) PINE is not limited to specific position embedding types.

###### Lemma 1 .

###### Theorem 1 .

### 3.4 Discussion

Different Position Re-Assignment Methods. PINE puts documents with higher importance scores to a closer position to queries. Another option is to put documents with higher importance scores in a more distant position to the queries. Considering the recency bias brought by the most popular rotary position embedding (RoPE) [], this alternative approach makes RoPE “disrespect" the attention of models. Therefore, we believe this alternative choice is not optimal, which is justified by our experiments in Section 4.3.

Figure: Figure 3: Previous work PCW [] eliminates position bias by first masking all inter-document attention and then re-assigning all documents the same positions. The notions are kept the same as Figure 2. Our experiment in Section 4 shows that PCW brings severe performance drop for tasks requiring language generation.
Refer to caption: x3.png

Different Attention Masks. Previous work PCW [] adopts a different way: it masks the inter-document attention instead of making it bidirectional (Figure 3, middle and right). Accordingly, it adopts a simplified position re-assignment method of ours: putting all documents in the same positions. However, masking all inter-document attention loses contextual information (the white part surrounded by colored blocks in Figure 3). Moreover, some different tokens now share the same positions (Figure 3, right), which could confuse models. As a result, PCW performs poorly in language generation tasks (Section 4).

Inference Cost. PINE incurrs additional computation overhead due to extra operations. Practically, the extra big $\mathcal{O}$ computation complexity to obtain hidden states is $\mathcal{O}(nk\log k)$, where $n$ and $k$ denote text length and the number of input documents, respectively. The bidirectional attention does not bring extra cost, the position re-assignment brings $\mathcal{O}(k\log k)$ for each token since the sorting algorithms are involved. The real computation cost is acceptable since $k$ is usually small (e.g., $k=2$ in the LLM-as-a-judge task and $k=20$ in the retrieval-augmented QA). Section 4.5 shows results of real-world wall time and memory cost.

## 4 Experiment

Our experiments aim to show PINE can improve model performance across diverse tasks and have superior performance than other approaches.

### 4.1 Settings

We select four tasks that pose position bias: LM-as-a-judge [] that prompts LMs to select a better response out of two given a question, retrieval-augmented question-answering [] that asks LMs to answer questions based on retrieved documents, molecule generation based on provided properties [], and math reasoning based on several given conditions []. We follow previous work [, ] and use temperature 0 in avoid variance.

LM-as-a-judge. We benchmark our method on 23 datasets in the RewardBench(^3^33Apache-2.0 license. https://github.com/allenai/reward-bench) [] that can be categorized into four types: Chat, Chat-Hard, Safety, and Reasoning. We use the official data split, prompts, and evaluation scripts to ensure reproducibility. We use LLaMa-3-Instruct models [] and Qwen-1.5-Chat models [] for experiments. To show how positions affect results, we present four results: the ground-truth response is positioned at first, second, or shuffled, and PINE results (which yield the same results for all three scenarios above).

Retrieval-augmented QA. We follow the settings and use the prompts, data, and evaluation scripts of [](^4^44MIT license. https://github.com/nelson-liu/lost-in-the-middle): Only one of the retrieved documents (10 or 20 in total) contains the ground-truth answer for the given question. We list prompts in Appendix D. We use LLaMa-3-70B-Instruct model [] for experiment. To show how positions affect results, we present several results: the ground-truth document is positioned at the beginning, middle, last, or shuffled, and PINE results (which yield the same results for all scenarios above).

Molecule Generation. In this task, the input contains several properties that are interchangeable, and LMs are asked to generate molecules that satisfy these properties. We train such an LM with QM9 [] dataset. The QM9 dataset collects over $130k$ 3D molecules with 3D structures [] calculated by density functional theory (DFT). Each molecule in QM9 has less than 9 heavy atoms, and its chemical elements all belong to H, C, N, O, F. We take six quantum property values as the conditional input to LMs and train LMs to generate molecules with the conditioned quantum property values. We split the training dataset of QM9 to two subsets where each subset has $50k$ samples, and train LMs and an EGNN-based quantum property prediction models [] on these two subsets, respectively. The six quantum properties are polarizability ($\alpha$), HOMO energy ($\epsilon_{\text{HOMO}}$), LUMO energy ($\epsilon_{\text{LUMO}}$), HOMO-LUMO gap ($\Delta\epsilon$), dipole moment ($\mu$) and heat capacity at $298.15$K ($C_{v}$). The LM is a 8-layer Llama model with 8 attention heads and 768 hidden dimensions. To evaluate the performance, we sample 10000 sets of 6-property conditions, randomize the property order in each condition, and generate molecules conditioned on these property values by the trained LM, and compute the mean absolute difference (MAE) between the given property values and the property values of the generated molecules. Note that we use the trained EGNN-based property prediction models to calculate the property values of the generated molecules.

Math Reasoning. We use R-GSM [], a subset of GSM8K. This small dataset (which contains 220 problems) is designed to test LMs’ performance with interchangeable premise orders. Problems in the dataset contain several conditions that do not have a progressive relationship. Therefore, their positions are interchangeable. We further clean this dataset to remove problems where conditions do not read smoothly after changing positions (e.g., use pronouns in the first condition but introduce names in the second condition), yielding a small set containing 95 problems. We test Qwen-1.5 models on this dataset.

More details of the four tasks can be found in Appendix D. Qualitative examples of the four tasks can be found in Appendix E.

Baselines. The goal of PINE is to eliminate position bias during inference mechanically. Therefore, we choose methods that have the same design principle as our baselines: (1) Vanilla inference (2) Vanilla inference with no inter-document attention (NIA for short, i.e., the middle figure in Figure. 3): The latter documents will have no attention to formers. (3) Parallel Context Window (PCW, rightmost in Figure. 3) []: PCW extends the baseline (2) by manipulating positions of documents. PCW allows all documents to share the same positions. (4) Structured Prompting (SP, a variant version of PCW) []: SP extends (3) by lowering attentions between decoded tokens and input documents to $\frac{1}{N}$ to solve the perplexity exploding problem in PCW. Similar to the proof in Section 3.3, we can know that (1) and (2) are not inter-document position invariant, whereas (3) and (4) are. Beyond these methods, we also introduce two other debiasing baselines: permutation [] and calibration [].

**Table 1: The portion of data (%) that models have position bias in RewardBench, i.e., models change answers after swaping candidate responses orders. We color the subsets that have more than $25\%$ data causing position bias with cyan.**
| Model | Size | Chat | Chat-Hard | Safety | Reasoning | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| LLaMa-3<br>-Instruct | 8B | $10.3$ | $21.5$ | $11.4$ | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}27.6}$ | $17.7$ |
| 70B | $3.6$ | $16.0$ | $5.8$ | $15.2$ | $10.2$ |  |
| Qwen-1.5<br>-Chat | 1.8B | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}33.5}$ | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}37.9}$ | $24.7$ | $13.3$ | $27.4$ |
| 4B | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}48.0}$ | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}38.6}$ | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}57.4}$ | $12.7$ | $39.2$ |  |
| 7B | $17.0$ | $20.6$ | $10.9$ | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}26.5}$ | $18.8$ |  |
| 32B | $7.8$ | $20.0$ | $9.6$ | ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}<br>\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}26.4}$ | $16.0$ |  |
| 72B | $10.9$ | $22.6$ | $9.6$ | $24.7$ | $17.0$ |  |
| 110B | $8.7$ | $16.0$ | $11.5$ | $23.5$ | $14.9$ |  |

Table: Table 2: Main results of RewardBench. Vanilla denotes the normal inference, (GT at A) means the ground truth chosen response is presented at the first, and (GT at B) indicates the second. For the 72B model, we additionally benchmark the Qwen 2.5 model. PINE consistently improves LM’s performance across different models and sizes and is particularly useful when assessing reasoning pairs.

**Table 3: Baseline performance on RewardBench. PINE achieves superior performance to baseline models, performing $4.8\%$ and $4.7\%$ better than the best performed baseline on two models.**
| Method | LLaMa-3-8B-Instruct | Qwen1.5-7B-Chat |  |  |
| --- | --- | --- | --- | --- |
| Reasoning | Full Set | Reasoning | Full Set |  |
| NIA (GT at A) | $43.7$ | $56.3$ | $60.7$ | $61.3$ |
| NIA (GT at B) | $66.7$ | $65.8$ | $44.1$ | $52.2$ |
| NIA | $55.9$ | $61.9$ | $51.4$ | $56.8$ |
| PCW | $56.5$ | $61.7$ | $53.4$ | $55.2$ |
| SP | $55.4$ | $60.8$ | $52.4$ | $55.4$ |
| PINE | $\bm{73.4}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+16.9}}}$ | $\bm{66.7}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+4.8}}}$ | $\bm{63.0}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+9.6}}}$ | $\bm{61.5}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+4.7}}}$ |

### 4.2 Results on LM-as-a-judge

Position bias exists across different models and sizes. Table 1 shows the statistics of position bias in RewardBench with different models. Position bias is quite common in RewardBench, and can be up to $48.0\%$. Larger models have less position bias, however, the position bias could still on average affect up to $10\%$ data.

PINE consistantly improve model performance across models and sizes. Table 2 shows the main results on RewardBench. We experiment with Llama-3 and Qwen-1.5 across different model sizes. The position of the ground truth chosen option is randomly shuffled. Therefore, the accuracy of the random guess method is expected to be $50\%$. First, the first two rows reveal that larger models tend to have a primacy bias, whereas smaller models tend to have a recency bias. By comparing the last two rows of each model size, we conclude that models across different sizes perform better with the help of PINE by eliminating position bias. The only exception is the Qwen-1.5-72B-Chat model. We suspect this model is not well-trained since Qwen-1.5-32B-Chat performs the same as the 72B model in vanilla inference, despite half of the model size. Qwen 2 report [] also shows that the Qwen 1.5B 72B model performs even worse than 32B in reasoning. Moreover, Table 2 shows that Qwen 2.5 72B can obtain consistent performance gains. Overall, PINE improves performance from a statistical perspective and makes models more reliable when as evaluators. Full results are shown in Appendix C.

PINE is extremely useful when assessing reasoning pairs. PINE consistently improves model performance on the “reasoning" subset by a large margin: from $8$ to $10$ percentage points in most cases. Speficially, LlaMa-3 Instruct 70B is originally is ranked 22nd generative model in the reasoning subset of RewardBench. With PINE, it achieves the 7th rank ($87.6\%$), outperforming GPT-4-0125-preview (the previous 8th rank, $86.9\%$), GPT-4o-2024-08-06 (the previous 9th rank, $86.6\%$), and Llama-3.1-405B-Instruct-Turbo (the previous 7th rank, $87.1\%$).(^5^55Results are provided by the official leaderboard (as of Sep 17, 2024): [https://huggingface.co/spaces/allenai/reward-bench](https://huggingface.co/spaces/allenai/reward-bench)).

PINE performs better than baseline models that adopt different attention masks. We then compare PINE with baseline models mentioned in Section 4.1 on Llama-3-8B-Instruct and Qwen1.5-7B-Chat model. They adopt a different attention mask: masking inter-document attention instead of making them bi-directional. Since NIA is not inter-document position-invariant, we also apply NIA with two extreme cases: the ground truth chosen response is always in the first or second place. Results on Table 3 show that PINE achieves the best performance and largely outperforms the best baselines by $\sim 5\%$, and outperforms NIA even if NIA is placed in the extreme case. On the reasoning subset, this performance gap becomes much even greater. The results reveal that masking inter-document attention mask is much less effective than bidirectional inter-document attention mask applied in PINE.

Another two widely used debiasing methods are permutation [] and calibration []. They are usually used in the logit-based evaluation or single-token generation. Their effectiveness in the open-ended generation is less explored. In our experiments, we find calibration methods generates rubbish responses, which we believe is because of the strong assumption in []: uniform distribution of all tokens in the generation task. For the permutation methods, we find LLama-3-8B-Instruct have $69.0\%$ and $65.9\%$ accuracy, Qwen1.5-7B-Chat has $58.2\%$ and $61.3\%$ accuracy on the reasoning set and fullest respectively, all underperforming PINE (numbers reported in Table 3).

### 4.3 Results on Retrieval-Augmented Question-Answering

Figure: Figure 4: The results of retrieval-augmented QA on Llama-3-70B-Instruct. Dashed lines indicate that the method is either inter-document position-invariant or the result is obtained on the order-shuffled data (denoted in the legend). (a) shows results of PINE against baselines. (b) shows results of different designs of PINE.
Refer to caption: x4.png

PINE performs better than baselines, on-par with vanilla inference on average while not being affected by the worst case. Models tend to perform better when the gold-standard document is at the beginning and the end of all documents in retrieval-augmented question-answers. Figure 4 (a) shows the results on LLaMa-3-70B-Instruct when 10 or 20 documents were presented. First, it is easy to conclude that all baselines are much worse than PINE (the pink line), which is consistent to the previous experiment. Second, PINE  achieves on-par performance on average compared with vanilla inference while being inter-document position invariant. Specifically, PINE  is slightly better/worse than vanilla inference with the gap +1.2/-2.0 when there are 10 and 20 documents in total. We hypothesize that the slight performance drop of PINE  for the 20 document setting is due to the performance drop of document importance score computation in PINE when presented with many documents. However, PINE is position-invariant, therefore does not be affected by the worst case (the bottom of blue solid curves). Third, the height generally becomes smaller between blue and brown solid lines in Figure 4 (a), and between the blue and red solid lines in Figure 4 (b) when the gold-standard document position increases, reflecting the causal attention generally prefers distant content, which is consistent to the hypothesis in Section 3.2. The brown line in Figure 4 (a) and red line (b) generally reflect recency bias brought by RoPE, which is consistent to previous works [, ].

PINE performs better than other position assignment methods.
So far, our experiments show that bidirectional inter-document attention is the better design choice than the masked one. However, there are still several design options for the position assignment, as discussed in Section 3.4. The first option is to re-assign position reversely, and the other is to use PINE without position re-assignment (i.e., use input document positions when they serve as keys). To gain a deeper understanding, we extend the retrieval-augmented QA experiments with the two mentioned alternative position assignment methods, and the results are presented in Figure 4 (b). The figure tells us that PINE is slightly better than PINE  without position re-assignment on average (+0.3. The gap becomes larger when 20 documents are presented: +1.5). Position re-assignment reversely has relatively worse results, showing that PINE is a better design choice, which is consistent with the intuitive analysis mentioned in Section 3.4. Although position re-assignment seems only to bring less gains than bidirectional attention mask, it is required to complete the proof that PINE can eliminate the position bias. Therefore, PINE without position re-assignment may suffice if one does not aim to eliminate the position bias and cares more about efficiency (no extra $\mathcal{O}(nk\log k)$ sorting cost).

**Table 4: The result of molecule generation on QM9 dataset. PINE improves model performance in 5 out of 6 criteria.**
| Model | $\alpha$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ | $\mu$ | $C_{v}$ |
| --- | --- | --- | --- | --- | --- | --- |
| LLama | $6.3997$ | $103.93$ | $53.4$ | $99.13$ | $\bm{3.4112}$ | $4.3785$ |
| Llama + PINE | $\bm{6.3702}$ | $\bm{102.15}$ | $\bm{53.09}$ | $\bm{98.27}$ | $3.4917$ | $\bm{4.2886}$ |

### 4.4 Results on Molecule Generation and Math Reasoning

PINE improves model performance on 5 out of 6 criteria in molecule generation . Table 4 shows the results of molecule generation. The consistent gain in 5 out 6 criteria shows the effectiveness of PINE.

Figure: Figure 5: Math reasoning results of Qwen1.5 series on R-GSM subset. PINE improves the reasoning accuracy by $12.6\%$ and $5.3\%$ with 7B and 110B models respectively compared with vanilla inference.
Refer to caption: x5.png

PINE improves math reasoning capabilities. Figure 5 shows the results of Qwen1.5 models on R-GSM dataset. It can be shown that PINE outperforms vanilla inference for both small 7B models and large 110B models.

### 4.5 Computational Overhead

Section 3.4 briefly discusses the computational overhead, with a conclusion that PINE ’s efficiency is still doable. In our experiments, we find the wall time of PINE is $\sim$2x and $\sim$8x of the vanilla inference on the LM-as-a-judge task and retrieval-augmented QA task with 20 documents, which is acceptable at least during experiments. However, we did not specially optimize codes to accelerate PINE, and our implementation still contains a “for” loop. Therefore, we believe there is room to accelerate PINE. Compared with the time overhead, the memory overhead is small and PINE can be run with 70B models on 3x A100 80G on the retrieval-augmented QA task, which requires the same number of GPUs as the vanilla inference. Since efficiency is not the main focus of this paper, we leave this as our future work.

## 5 Conclusion, Limitations and Future Work

This paper logically analyzes the transformers and attributes the position bias to causal attention and position embedding. Based on the analysis, we propose a novel train-free zero-shot approach to eliminate the position bias mechanically. The core idea is to make every input documents equally affected by the attention mask and position embedding. Experiments on four tasks show the effectiveness of our approach. However, PINE requires extra computation, and thus it is not suitable for scenarios that are efficiency-sensitive. We believe there is room to improve the efficiency with more efficient implementation, and we leave this as our future work.

## Acknowledgement

We thank Chujie Zheng for the helpful discussion and feedback.

## Appendix A Another Example of Position Bias in VLMs

Figure: Figure 6: Image of Andromeda Galaxy. GPT-4V is asked to answer the galaxy of the redbox (M110). (b), (c), (d) images are generated by flipping (a). Results show that GPT-4V can answer correctly on (c) and (d) and incorrectly on (a) and (b), revealing position bias in VLMs.
Refer to caption: x6.png

To show VLMs suffer from position bias, we ask models to identify a small object appeared in the full image. We ask GPT-4V [] to identify one satellite galaxy M110 of Andromeda Galaxy (M31). The central objective in the Figure 6 (a) is the Andromeda Galaxy and the galaxy in the red box is M110 satellite galaxy. Another satellite galaxy is M32 which is the brightest part near the Andromeda Galaxy.

We flip the image (a) and obtains images (b), (c), and (d). We prompt GPT-4V “What galaxy is in the red box?”, and here are the responses we collected:

(a): The galaxy within the red box is M32, also known as Messier 32. It’s a dwarf elliptical galaxy and is a satellite of the Andromeda Galaxy (M31), which is the prominent galaxy in the image. M32 is one of the more well-studied dwarf elliptical galaxies, largely due to its proximity to our own galaxy and its interactions with the Andromeda Galaxy.

(b): The object inside the red box in the image appears to be a smaller galaxy located near the larger, more prominent galaxy (Andromeda Galaxy, M31). This smaller galaxy is likely M32, a satellite galaxy of the Andromeda Galaxy. M32 is a compact elliptical galaxy and is often visible in images as a smaller, dense, bright spot near Andromeda, as seen in the boxed area of your image.

(c): The galaxy within the red box is M110, also known as NGC 205. M110 is a dwarf elliptical galaxy, which, like M32, is a satellite of the Andromeda Galaxy (M31). It is one of the members of the Local Group of galaxies, which includes the Milky Way, the Andromeda Galaxy, and about 54 other known galaxies.

(d):
The galaxy in the red box appears to be a satellite galaxy of the Andromeda Galaxy, which is the prominent galaxy visible in the rest of the image. This smaller galaxy is likely M110 (NGC 205), which is a dwarf elliptical galaxy and a companion to the Andromeda Galaxy, M31. It’s one of the several satellite galaxies gravitationally bound to Andromeda, visible here as a faint, elongated object in the outlined area.

We can find that models answer corrected when M110 is at the top of the image, revealing that VLMs also suffer from the position bias. The position bias may lead unreliable VLMs when fine-grained image analysis are needed (e.g., small object detection []).

## Appendix B Complete Proof

This section provided a complete proof to show PINE can eliminate position bias.

To simplify the notation and without loss of generality (w.l.o.g), we still use examples in Section 3.1.

###### Lemma 1 .

If the input $\mathbf{Q},\mathbf{K},\mathbf{V}$ are inter-document position-invariant representations, then $\mathbf{H}_{\text{PINE}}$ are also inter-document position-invariant representations.

Proof: First, the SYS tokens already satisfy this lemma under the vanilla inference since they appear before documents, and PINE does not change their computation process. We only need to show PINE can make $\mathcal{D}_{i}$ and Token 8 (i.e., tokens after documents) satisfy the lemma. W.l.o.g, we use $\mathcal{D}_{1}$ as a running example:

- •
PINE first obtains importance score between documents: $\text{Sim}(\mathcal{D}_{1},\mathcal{D}_{i})=\sum\text{Softmax}(\textbf{Q}_{1}
\textbf{K}_{i}^{T}/\sqrt{(}d))/|\mathcal{D}_{i}||$, where $\textbf{Q}_{1}\in\mathbb{R}^{2\times d},\textbf{K}_{i}\in\mathbb{R}^{2\times d}$, 2 denotes the number of tokens in documents, and $d$ denotes hidden states dimensions. Note that here the $\textbf{Q},\textbf{K}$ have not been applied to position embedding yet. Therefore, the importance score is not a function of input document positions.
- •
W.l.o.g, let’s assume $\text{Sim}(\mathcal{D}_{1},\mathcal{D}_{2})>\text{Sim}(\mathcal{D}_{1},
\mathcal{D}_{3})$, then we sort the document as follows $[\mathcal{D}_{3}|\mathcal{D}_{2}|\mathcal{D}_{1}]$ when they serve as keys and $\mathcal{D}_{1}$ as query. Concretely, $\textbf{Q}_{\text{PE},1}=\text{PE}(\textbf{Q}_{1},3)$ (3 denotes it is treated as the last, i.e., third, document), $\textbf{K}_{\text{PE},1}=\text{PE}(\textbf{K}_{1},3)$, $\textbf{K}_{\text{PE},2}=\text{PE}(\textbf{K}_{2},2)$, $\textbf{K}_{\text{PE},3}=\text{PE}(\textbf{K}_{3},1)$. Then we compute hidden states of $\mathcal{D}_{1}$: $\textbf{H}_{1}=\text{Softmax}(\textbf{Q}_{\text{PE},1}\textbf{K}_{\text{PE}}/
\sqrt{(}d))$, where $\textbf{K}_{\text{PE}}$ is the key values for the whole sequence $[\text{SYS}|\mathcal{D}_{3}|\mathcal{D}_{2}|\mathcal{D}_{1}]$. It is noted that this process does not use any variables that are dependent on the input document positions, nor directly use the input document positions. Therefore, $\textbf{H}_{1}$ obtained by PINE is not a function of input document positions.
- •
Similarly, $\textbf{H}_{2}$, $\textbf{H}_{3}$, and Token 8’s hidden states are not functions of input document positions. Their concatenation yields $\textbf{H}_{\text{PINE}}$, which is not a function of input document positions.

Proof ends.

###### Theorem 1 .

Given an input, if $\mathbf{H}_{\text{PINE}}$ is applied to every layer, attention head, and token to replace the conventional attention computation, then the model outputs are inter-document position-invariant representations.

First, the embedding layer is not a function of input documents positions. Suppose that the $i$th layer’s input hidden states are not a function of input documents positions, then within each layer:

- •
The attention hidden states are not a function of input documents positions (Lemma).
- •
The Layernorm, FFN outputs are not a function of input documents positions.
- •
Therefore, the output hidden states of $i$th transformer layer, i.e., the input hidden states of $i+1$th transformer layer, are not a function of input documents positions.

Using mathematical induction, we know the final outputs are not a function of input documents positions.

Proof ends.

Notes on the proof:

- •
PINE needs to be applied on each layer, attention heads, and tokens to satisfy the above proof.
- •
The extra big $\mathcal{O}$ computation cost is purely come from the position re-assignment step: $\mathcal{O}(klogk)$ for sorting $k$ documents. Since we need to repeat this step for every token, the extra computation cost is $\mathcal{O}(nklogk)$, where $n$ is the number of tokens.
- •
Although position re-assignment brings an extra computational cost, it is a must to complete the proof. Removing this step will make PINE unable to “eliminate” position bias. Similarly, a bidirectional attention mask is also a must to complete the proof.
- •
PINE is not limited to specific position encoding algorithms.

###### Lemma 1 .

###### Theorem 1 .

## Appendix C Full results of RewardBench

Table 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 present the full results of the reward bench. After inspecting the error cases, we categorize the performance drop in the Chat-Hard and Safety subsets into two main aspects:

- •
The instruction-following capabilities become a bit worse. For example, LLMs tend to solve the “user question” instead of comparing two responses, or LLMs do not output answers in requested formats, causing parsing failures when computing performance scores.
- •
LMs overly focus on helpfulness in safety prompts, therefore causing performance degradation in the Safety dataset.

However, the positive effect of PINE (i.e., eliminating position bias) is more significant than these negative effects; therefore, the overall PINE is still beneficial to models.

Table: Table 5: Full results of Table 2. Vanilla denotes the normal inference, (GT at A) means the ground truth chosen response is presented at the first, and (GT at B) indicates at the second. PINE consistently improves LM’s performance across different model sizes. Consistent to Table 1, we color the subsets with severe position bias cyan. It can be observed that PINE generally improves performance on cyan subsets by a large margin, which is consistent to our motivation and goal.

**Table 6: Full version of Table 3. PINE achieves superior performance to baseline models, performing $4.8\%$ and $4.7\%$ better than the best performed baseline on two models.**
| Model | Method | Chat | Chat-Hard | Safety | Reasoning | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| LLaMa-3<br>8B-Instruct | NIA (GT at A) | $81.0$ | $40.7$ | $59.7$ | $43.7$ | $56.3$ |
| NIA (GT at B) | $81.0$ | $49.7$ | $65.8$ | $66.7$ | $65.8$ |  |
| NIA | $80.9$ | $46.7$ | $64.0$ | $55.9$ | $61.9$ |  |
| PCW | $78.6$ | $46.8$ | $64.8$ | $56.5$ | $61.7$ |  |
| SP | $79.6$ | $43.3$ | $65.0$ | $55.4$ | $60.8$ |  |
| PINE | $85.6$ | $41.5$ | $66.5$ | $73.4$ | $\bm{66.7}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+4.8}}}$ |  |
| Qwen-1.5<br>7B-Chat | NIA (GT at A) | $67.7$ | $57.2$ | $59.6$ | $60.7$ | $61.3$ |
| NIA (GT at B) | $67.9$ | $35.9$ | $61.0$ | $44.1$ | $52.2$ |  |
| NIA | $74.9$ | $43.5$ | $57.4$ | $51.4$ | $56.8$ |  |
| PCW | $67.2$ | $42.0$ | $58.3$ | $53.4$ | $55.2$ |  |
| SP | $69.4$ | $41.8$ | $58.0$ | $52.4$ | $55.4$ |  |
| PINE | $85.8$ | $38.7$ | $58.6$ | $63.0$ | $\bm{61.5}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+4.7}}}$ |  |

**Table 7: Meta-Llama-3-8B-Instruct results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $\bm{91.0}$ | $90.0$ |
| alpacaeval-hard | $\bm{91.6}$ | $90.0$ |
| alpacaeval-length | $71.6$ | $\bm{77.4}$ |
| donotanswer | $\bm{45.2}$ | $38.2$ |
| hep-cpp | $78.7$ | $\bm{82.6}$ |
| hep-go | $77.1$ | $\bm{86.6}$ |
| hep-java | $73.5$ | $\bm{82.9}$ |
| hep-js | $74.4$ | $\bm{84.1}$ |
| hep-python | $79.0$ | $\bm{85.7}$ |
| hep-rust | $74.4$ | $\bm{81.4}$ |
| llmbar-adver-GPTInst | $23.4$ | $\bm{24.5}$ |
| llmbar-adver-GPTOut | $63.8$ | $\bm{67.0}$ |
| llmbar-adver-manual | $\bm{40.2}$ | $34.8$ |
| llmbar-adver-neighbor | $\bm{20.9}$ | $16.8$ |
| llmbar-natural | $66.0$ | $\bm{74.5}$ |
| math-prm | $54.5$ | $\bm{62.8}$ |
| mt-bench-easy | $\bm{92.9}$ | $87.5$ |
| mt-bench-hard | $\bm{68.9}$ | $64.9$ |
| mt-bench-med | $\bm{83.8}$ | $80.0$ |
| refusals-dangerous | $71.5$ | $\bm{74.0}$ |
| refusals-offensive | $\bm{76.0}$ | $73.5$ |
| xstest-should-refuse | $70.5$ | $\bm{71.8}$ |
| xstest-should-respond | $72.0$ | $\bm{76.4}$ |

**Table 8: Meta-Llama-3-70B-Instruct results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $\bm{100.0}$ | $\bm{100.0}$ |
| alpacaeval-hard | $\bm{100.0}$ | $\bm{100.0}$ |
| alpacaeval-length | $\bm{91.1}$ | $89.5$ |
| donotanswer | $47.1$ | $\bm{48.2}$ |
| hep-cpp | $\bm{92.7}$ | $92.1$ |
| hep-go | $89.9$ | $\bm{97.0}$ |
| hep-java | $92.1$ | $\bm{97.0}$ |
| hep-js | $93.3$ | $\bm{95.1}$ |
| hep-python | $90.9$ | $\bm{95.4}$ |
| hep-rust | $89.0$ | $\bm{91.5}$ |
| llmbar-adver-GPTInst | $55.4$ | $\bm{57.6}$ |
| llmbar-adver-GPTOut | $73.4$ | $\bm{76.6}$ |
| llmbar-adver-manual | $\bm{53.3}$ | $47.8$ |
| llmbar-adver-neighbor | $\bm{32.8}$ | $28.7$ |
| llmbar-natural | $83.0$ | $\bm{84.0}$ |
| math-prm | $66.4$ | $\bm{80.5}$ |
| mt-bench-easy | $\bm{100.0}$ | $\bm{100.0}$ |
| mt-bench-hard | $\bm{78.4}$ | $75.7$ |
| mt-bench-med | $\bm{97.5}$ | $\bm{97.5}$ |
| refusals-dangerous | $\bm{63.5}$ | $62.5$ |
| refusals-offensive | $\bm{66.5}$ | $\bm{66.5}$ |
| xstest-should-refuse | $\bm{68.8}$ | $63.3$ |
| xstest-should-respond | $\bm{96.8}$ | $96.4$ |

**Table 9: Qwen1.5-1.8B-Chat results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $\bm{47.5}$ | $17.0$ |
| alpacaeval-hard | $\bm{56.3}$ | $13.7$ |
| alpacaeval-length | $\bm{50.0}$ | $45.8$ |
| donotanswer | $\bm{54.0}$ | $52.6$ |
| hep-cpp | $49.4$ | $\bm{51.5}$ |
| hep-go | $\bm{54.3}$ | $48.8$ |
| hep-java | $\bm{52.7}$ | $49.4$ |
| hep-js | $\bm{48.2}$ | $47.6$ |
| hep-python | $49.4$ | $\bm{52.1}$ |
| hep-rust | $\bm{54.6}$ | $50.3$ |
| llmbar-adver-GPTInst | $44.0$ | $\bm{76.6}$ |
| llmbar-adver-GPTOut | $\bm{55.3}$ | $40.4$ |
| llmbar-adver-manual | $44.6$ | $\bm{64.1}$ |
| llmbar-adver-neighbor | $56.7$ | $\bm{66.8}$ |
| llmbar-natural | $49.0$ | $\bm{51.5}$ |
| math-prm | $45.5$ | $\bm{70.2}$ |
| mt-bench-easy | $39.3$ | $\bm{57.1}$ |
| mt-bench-hard | $\bm{54.1}$ | $35.1$ |
| mt-bench-med | $\bm{46.2}$ | $45.0$ |
| refusals-dangerous | $48.5$ | $\bm{88.0}$ |
| refusals-offensive | $49.5$ | $\bm{54.5}$ |
| xstest-should-refuse | $53.2$ | $\bm{53.9}$ |
| xstest-should-respond | $52.2$ | $\bm{68.8}$ |

**Table 10: Qwen1.5-4B-Chat results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $59.5$ | $\bm{77.5}$ |
| alpacaeval-hard | $62.1$ | $\bm{80.5}$ |
| alpacaeval-length | $60.5$ | $\bm{70.0}$ |
| donotanswer | $\bm{54.4}$ | $18.4$ |
| hep-cpp | $\bm{50.0}$ | $\bm{50.0}$ |
| hep-go | $50.3$ | $\bm{51.8}$ |
| hep-java | $49.1$ | $\bm{51.2}$ |
| hep-js | $\bm{49.7}$ | $49.4$ |
| hep-python | $50.0$ | $\bm{53.0}$ |
| hep-rust | $\bm{50.6}$ | $\bm{50.6}$ |
| llmbar-adver-GPTInst | $36.4$ | $\bm{38.6}$ |
| llmbar-adver-GPTOut | $\bm{54.3}$ | $51.1$ |
| llmbar-adver-manual | $\bm{51.1}$ | $39.1$ |
| llmbar-adver-neighbor | $\bm{52.2}$ | $42.5$ |
| llmbar-natural | $48.0$ | $\bm{53.5}$ |
| math-prm | $58.2$ | $\bm{70.9}$ |
| mt-bench-easy | $44.6$ | $\bm{75.0}$ |
| mt-bench-hard | $\bm{58.1}$ | $48.6$ |
| mt-bench-med | $\bm{56.2}$ | $50.0$ |
| refusals-dangerous | $\bm{43.0}$ | $31.0$ |
| refusals-offensive | $47.0$ | $\bm{71.0}$ |
| xstest-should-refuse | $53.6$ | $\bm{64.3}$ |
| xstest-should-respond | $51.0$ | $\bm{71.0}$ |

**Table 11: Qwen1.5-7B-Chat results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $74.0$ | $\bm{91.0}$ |
| alpacaeval-hard | $90.0$ | $\bm{96.8}$ |
| alpacaeval-length | $65.8$ | $\bm{74.7}$ |
| donotanswer | $\bm{19.9}$ | $11.0$ |
| hep-cpp | $60.4$ | $\bm{76.8}$ |
| hep-go | $61.9$ | $\bm{69.2}$ |
| hep-java | $56.1$ | $\bm{74.1}$ |
| hep-js | $59.5$ | $\bm{68.6}$ |
| hep-python | $63.4$ | $\bm{66.8}$ |
| hep-rust | $62.8$ | $\bm{65.9}$ |
| llmbar-adver-GPTInst | $\bm{40.2}$ | $23.4$ |
| llmbar-adver-GPTOut | $\bm{53.2}$ | $45.7$ |
| llmbar-adver-manual | $\bm{35.9}$ | $\bm{35.9}$ |
| llmbar-adver-neighbor | $\bm{21.6}$ | $19.8$ |
| llmbar-natural | $\bm{71.0}$ | $70.0$ |
| math-prm | $\bm{57.9}$ | $55.8$ |
| mt-bench-easy | $\bm{87.5}$ | $85.7$ |
| mt-bench-hard | $\bm{62.2}$ | $55.4$ |
| mt-bench-med | $\bm{77.5}$ | $72.5$ |
| refusals-dangerous | $\bm{49.0}$ | $40.5$ |
| refusals-offensive | $\bm{86.0}$ | $\bm{86.0}$ |
| xstest-should-refuse | $\bm{74.0}$ | $63.6$ |
| xstest-should-respond | $75.6$ | $\bm{86.6}$ |

**Table 12: Qwen1.5-32B-Chat results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $\bm{97.0}$ | $\bm{97.0}$ |
| alpacaeval-hard | $\bm{98.9}$ | $\bm{98.9}$ |
| alpacaeval-length | $81.1$ | $\bm{82.1}$ |
| donotanswer | $\bm{44.5}$ | $41.2$ |
| hep-cpp | $87.8$ | $\bm{91.5}$ |
| hep-go | $80.5$ | $\bm{93.9}$ |
| hep-java | $88.7$ | $\bm{96.3}$ |
| hep-js | $84.5$ | $\bm{95.7}$ |
| hep-python | $86.3$ | $\bm{93.6}$ |
| hep-rust | $82.0$ | $\bm{88.1}$ |
| llmbar-adver-GPTInst | $\bm{43.5}$ | $34.8$ |
| llmbar-adver-GPTOut | $\bm{68.1}$ | $57.4$ |
| llmbar-adver-manual | $32.6$ | $\bm{37.0}$ |
| llmbar-adver-neighbor | $25.0$ | $\bm{28.4}$ |
| llmbar-natural | $83.0$ | $\bm{85.0}$ |
| math-prm | $48.7$ | $\bm{60.3}$ |
| mt-bench-easy | $\bm{96.4}$ | $92.9$ |
| mt-bench-hard | $\bm{81.1}$ | $75.7$ |
| mt-bench-med | $92.5$ | $\bm{95.0}$ |
| refusals-dangerous | $\bm{80.0}$ | $\bm{80.0}$ |
| refusals-offensive | $\bm{99.0}$ | $\bm{99.0}$ |
| xstest-should-refuse | $\bm{90.6}$ | $89.3$ |
| xstest-should-respond | $84.4$ | $\bm{85.6}$ |

**Table 13: Qwen1.5-72B-Chat results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $\bm{98.0}$ | $\bm{98.0}$ |
| alpacaeval-hard | $97.4$ | $\bm{97.9}$ |
| alpacaeval-length | $\bm{85.3}$ | $84.2$ |
| donotanswer | $\bm{39.0}$ | $38.2$ |
| hep-cpp | $88.1$ | $\bm{89.6}$ |
| hep-go | $85.4$ | $\bm{92.1}$ |
| hep-java | $87.2$ | $\bm{90.9}$ |
| hep-js | $\bm{90.9}$ | $\bm{90.9}$ |
| hep-python | $87.2$ | $\bm{89.6}$ |
| hep-rust | $\bm{88.1}$ | $87.2$ |
| llmbar-adver-GPTInst | $\bm{44.6}$ | $33.7$ |
| llmbar-adver-GPTOut | $57.4$ | $\bm{61.7}$ |
| llmbar-adver-manual | $\bm{41.3}$ | $39.1$ |
| llmbar-adver-neighbor | $\bm{28.0}$ | $20.9$ |
| llmbar-natural | $\bm{84.0}$ | $81.0$ |
| math-prm | $\bm{48.5}$ | $47.9$ |
| mt-bench-easy | $96.4$ | $\bm{100.0}$ |
| mt-bench-hard | $\bm{70.3}$ | $62.2$ |
| mt-bench-med | $\bm{95.0}$ | $92.5$ |
| refusals-dangerous | $\bm{75.5}$ | $73.0$ |
| refusals-offensive | $94.0$ | $\bm{95.0}$ |
| xstest-should-refuse | $87.7$ | $\bm{91.2}$ |
| xstest-should-respond | $\bm{86.8}$ | $85.0$ |

**Table 14: Qwen2.5-72B-Instruct results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $\bm{99.0}$ | $\bm{99.0}$ |
| alpacaeval-hard | $97.9$ | $\bm{98.9}$ |
| alpacaeval-length | $\bm{91.6}$ | $89.5$ |
| donotanswer | $48.5$ | $\bm{52.9}$ |
| hep-cpp | $\bm{95.7}$ | $\bm{95.7}$ |
| hep-go | $97.0$ | $\bm{98.8}$ |
| hep-java | $\bm{98.8}$ | $97.6$ |
| hep-js | $94.5$ | $\bm{98.2}$ |
| hep-python | $\bm{98.8}$ | $\bm{98.8}$ |
| hep-rust | $94.5$ | $\bm{97.6}$ |
| llmbar-adver-GPTInst | $66.3$ | $\bm{68.5}$ |
| llmbar-adver-GPTOut | $\bm{76.6}$ | $72.3$ |
| llmbar-adver-manual | $\bm{65.2}$ | $63.0$ |
| llmbar-adver-neighbor | $41.8$ | $\bm{44.0}$ |
| llmbar-natural | $\bm{92.0}$ | $87.0$ |
| math-prm | $74.5$ | $\bm{84.8}$ |
| mt-bench-easy | $\bm{100.0}$ | $\bm{100.0}$ |
| mt-bench-hard | $\bm{94.6}$ | $91.9$ |
| mt-bench-med | $97.5$ | $\bm{100.0}$ |
| refusals-dangerous | $78.0$ | $\bm{82.0}$ |
| refusals-offensive | $\bm{95.0}$ | $92.0$ |
| xstest-should-refuse | $\bm{92.2}$ | $89.6$ |
| xstest-should-respond | $\bm{95.2}$ | $93.6$ |

**Table 15: Qwen1.5-110B-Chat results on RewardBench**
| Dataset | Vanilla | PINE |
| --- | --- | --- |
| alpacaeval-easy | $95.0$ | $\bm{97.0}$ |
| alpacaeval-hard | $\bm{98.9}$ | $\bm{98.9}$ |
| alpacaeval-length | $\bm{93.7}$ | $88.4$ |
| donotanswer | $51.5$ | $\bm{55.9}$ |
| hep-cpp | $87.8$ | $\bm{92.1}$ |
| hep-go | $83.8$ | $\bm{94.8}$ |
| hep-java | $86.6$ | $\bm{94.8}$ |
| hep-js | $90.5$ | $\bm{92.4}$ |
| hep-python | $83.8$ | $\bm{93.9}$ |
| hep-rust | $85.7$ | $\bm{90.9}$ |
| llmbar-adver-GPTInst | $\bm{70.1}$ | $65.2$ |
| llmbar-adver-GPTOut | $\bm{72.3}$ | $61.7$ |
| llmbar-adver-manual | $60.9$ | $\bm{65.2}$ |
| llmbar-adver-neighbor | $\bm{44.8}$ | $41.4$ |
| llmbar-natural | $86.5$ | $\bm{90.0}$ |
| math-prm | $69.6$ | $\bm{79.2}$ |
| mt-bench-easy | $98.2$ | $\bm{100.0}$ |
| mt-bench-hard | $\bm{83.8}$ | $\bm{83.8}$ |
| mt-bench-med | $\bm{97.5}$ | $\bm{97.5}$ |
| refusals-dangerous | $76.0$ | $\bm{84.0}$ |
| refusals-offensive | $\bm{97.0}$ | $\bm{97.0}$ |
| xstest-should-refuse | $91.6$ | $\bm{91.9}$ |
| xstest-should-respond | $\bm{95.6}$ | $92.4$ |

## Appendix D Implementation Details

### D.1 Experiment Setting

For reproducibility, the generation temperature is set to 0. We use PyTorch [, ],(^6^66Customized license. https://github.com/pytorch/pytorch) Transformers [],(^7^77Apache-2.0 license. https://huggingface.co/docs/transformers/en/index) and vLLM [] for our experiments.(^8^88Apache-2.0 license. https://github.com/vllm-project/vllm.) All experiments are launched with a single node of 8x A100 80G with SXM connection. 70B and 110B models are launched with 3x and 4x A100, and other model sizes can be launched with 1x A100.

### D.2 Prompts

We use the prompts provided by RewardBench [] official repo for the LM-as-a-judge task:

System prompt:

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better.

User prompt:

[User Question]

......

[The Start of Assistant A’s Answer]

......

[The End of Assistant A’s Answer]

[The Start of Assistant B’s Answer]

......

[The End of Assistant B’s Answer]

We use the prompts of official repo [] for the retrieval-augmented QA experiments:

User Prompt:

Write a high-quality one-sentence answer for the given question using only the provided search results (some of which might be irrelevant).

Document (Title: ......): ......

Document (Title: ......): ......

......

Document (Title: ......): ......

Document (Title: ......): ......

Question: ......

Molecule generation does not need prompts. We use prompts in OpenAI/simple-evals (^9^99https://github.com/openai/simple-evals) to evaluate R-GSM dataset.

## Appendix E Qualitative Examples

### E.1 LM-as-a-judge

[User Question]

Who created the Superman cartoon character?

[The Start of Assistant A’s Answer]

The Superman cartoon character was created by Jerry Siegel and Joe Shuster in 1938.

[The End of Assistant A’s Answer]

[The Start of Assistant B’s Answer]

Superman, the iconic comic book superhero, was created by writer Jerry Siegel and artist Joe Shuster. Superman first appeared in Action Comics #1, which was published by Detective Comics, Inc. (later DC Comics) in June 1938. The character’s immense popularity established him as one of the most enduring and recognizable figures in the superhero genre.

[The End of Assistant B’s Answer]

Here are outputs of Qwen1.5-7B-Chat:

Vanilla: Both AI assistants provide accurate and informative responses to the user’s question. Assistant A’s answer is brief and directly states the creators’ names and the year of creation, while Assistant B offers additional context by mentioning the first appearance, the publisher, and the character’s significance in the superhero genre. However, since the user asked for the creators, Assistant A’s answer is more directly helpful in this specific context. Final verdict: [[A]]

Vanilla (Swapped): [[B]]. Assistant B’s response is more detailed and informative, providing additional context about the origin of the character in Action Comics #1, the publisher, and its significance in the superhero genre. It also includes the names of the creators, which is crucial information for the question. Assistant A’s response is shorter and lacks the historical context and specific details. Therefore, Assistant B is better in answering the user’s question.

PINE: [[B]]. Assistant B provides a more comprehensive and detailed response to the user’s question. It not only mentions the creators but also provides additional information about the character’s debut and the company that published the comic. This answer is not only accurate but also offers context that highlights the significance of the creation. Assistant A’s response is brief and lacks the depth and historical context provided by Assistant B.

### E.2 Retrieval-Augmented Question-Answering

User Prompt:

Document (Title: Pearl) south Pacific and Indian Ocean. The largest pearl oyster is the "Pinctada maxima", which is roughly the size of a dinner plate. South Sea pearls are characterized by their large size and warm luster. Sizes up to 14 mm in diameter are not uncommon. In 2013, Indonesia Pearl supplied 43 percent of South Sea Pearls international market. The other significant producers are Australia, Philippines, Myanmar and Malaysia. In 1914, pearl farmers began growing cultured freshwater pearls using the pearl mussels native to Lake Biwa. This lake, the largest and most ancient in Japan, lies near the city of Kyoto. The

Document (Title: Laccadive Sea) the gulf as most productive in the world. Although extraction of natural pearls is considered too expensive in most parts of the world, it is still conducted in the gulf. Also collected in large numbers are Shankha mollusks ("Xancus pyrum") whose shells are used as a ritual and religious object. Other mollusks of the sea are either too scarce or not popular in the Indian society and therefore have no commercial value. Another traditional occupation in the Laccadive Sea is fishing. The annual fish catch is 2,000 to 5,000 tonnes from the Lakshadweep islands, which is mostly constituted by tuna

Document (Title: Pearl) including the Cook Islands and Fiji are being extensively used for producing cultured pearls. The rarity of the black cultured pearl is now a "comparative" issue. The black cultured pearl is rare when compared to Chinese freshwater cultured pearls, and Japanese and Chinese akoya cultured pearls, and is more valuable than these pearls. However, it is more abundant than the South Sea pearl, which is more valuable than the black cultured pearl. This is simply because the black pearl oyster "Pinctada margaritifera" is far more abundant than the elusive, rare, and larger south sea pearl oyster "Pinctada maxima", which cannot

Document (Title: Pearl powder) Pearl powder Pearl powder () is a preparation of crushed pearls used in China and elsewhere for skin care and in traditional Chinese medicine. Pearl powder is made from freshwater pearls or saltwater pearls below jewellery grade. These are sterilised in boiling water and then milled into a fine powder using stainless steel grinding discs or by milling with small porcelain balls in moist conditions. The powder is sold as such or mixed into creams. Pearl powder is widely believed to help improve the appearance of the skin, and is used as a cosmetic by royal families in Asia. It

Document (Title: Hyderabad pearl) with white pearls. Recently, several pearl makers are exporting processed pearls to markets in Europe and the US. With the capital that they gain from this marketing, they are able to purchase machinery for advanced refinement. In particular, equipment that uses enzymes present in thermophiles is able to substantially improve the process of refining pearls. Hyderabad pearl Hyderabad is considered the main pearl trading center in India. The most notable area devoted to the trade is the village called Chandanpet just outside Hyderabad, wherein almost the entire population is engaged in the delicate art of drilling pearls, a skill they

Document (Title: Pearl) pearls". The correct definition of a South Sea pearl – as described by CIBJO and GIA – is a pearl produced by the "Pinctada maxima" pearl oyster. South Sea pearls are the color of their host "Pinctada maxima" oyster – and can be white, silver, pink, gold, cream, and any combination of these basic colors, including overtones of the various colors of the rainbow displayed in the pearl nacre of the oyster shell itself. South Sea pearls are the largest and rarest of the cultured pearls – making them the most valuable. Prized for their exquisitely beautiful órientór lustre,

Document (Title: Chandrani Pearls) year 2007–08 Chandrani Pearls imported their pearls from Japan, China or Korea. Chandrani Pearls Chandrani Pearls is a prominent pearl jewelery brand of India. It pioneered the concept of pearls in India. Chandrani Pearls’s headquarters is at Kolkata in West Bengal. Chandrani Pearls was started on 24 January 1985 by Mr. Kuldip Nayar, his wife Mrs. Lakshmi Nayar and his father late Mr. N.C. Nayar in Kolkata’s up market Minto Park area. Chandrani Pearls management is now assisted by Nisheeth Nayar, sons of Mr. Kuldip Nayar. Chandrani Pearls have 63 showrooms across 9 states. From a modest turnover of Rs.

Document (Title: Pearl) For thousands of years, seawater pearls were retrieved by divers in the Indian Ocean in areas such as the Persian Gulf, the Red Sea and the Gulf of Mannar. Evidence also suggest a prehistoric origin to pearl diving in these regions. Starting in the Han Dynasty (206 BC–220 AD), the Chinese hunted extensively for seawater pearls in the South China Sea. In the 14th-century Arabian Sea, the traveller Ibn Battuta provided the earliest known description of pearl diving by means of attaching a cord to the diver’s waist.

Document (Title: Chandrani Pearls) Chandrani Pearls Chandrani Pearls is a prominent pearl jewelery brand of India. It pioneered the concept of pearls in India. Chandrani Pearls’s headquarters is at Kolkata in West Bengal. Chandrani Pearls was started on 24 January 1985 by Mr. Kuldip Nayar, his wife Mrs. Lakshmi Nayar and his father late Mr. N.C. Nayar in Kolkata’s up market Minto Park area. Chandrani Pearls management is now assisted by Nisheeth Nayar, sons of Mr. Kuldip Nayar. Chandrani Pearls have 63 showrooms across 9 states. From a modest turnover of Rs. 2.75 lakhs in 1990 to a turnover of Rs. 15.00 crores in

Document (Title: Hyderabad pearl) Hyderabad pearl Hyderabad is considered the main pearl trading center in India. The most notable area devoted to the trade is the village called Chandanpet just outside Hyderabad, wherein almost the entire population is engaged in the delicate art of drilling pearls, a skill they have practiced for generations. This practice also makes the city one of the largest pearl drilling locations in India. The pearl industry in Hyderabad flourished due to the patronage of the Qutub Shahi kings and the Asaf Jahis, who were said to have an affinity for sparkling jewels. The pearls were not only part of

Question: in which sea pearl is found in india?

Here are Llama-3-70B-Instruct results:

Vanilla:
According to the search results, pearls are found in the Laccadive Sea in India.

PINE:
Pearls are found in the Indian Ocean, specifically in the Gulf of Mannar and the Laccadive Sea, off the coast of India.

### E.3 Molecule Generation

The task has properties as input, and molecule structure as output. Below is one example:

Input: cv*-30.887* alpha*81.68* mu*2.0332* homo*-5.8749* gap*7.8668* lumo*1.9919*

Output: H 0.00 0.00° 0.00° O 0.96 1.57° 0.00° C 1.95 2.33° 0.00° H 2.35 2.50° 0.70° H 2.32 2.53° -0.71° C 3.22 2.00° -0.02° H 3.46 1.80° 0.24° C 4.50 2.23° 0.00° H 4.99 2.22° 0.26° H 4.55 2.46° -0.12° C 5.58 2.08° -0.18° H 6.55 2.07° -0.09° N 5.89 2.05° -0.46° C 5.57 2.18° -0.71° H 4.57 2.25° -0.78° H 6.29 2.32° -0.72° H 5.98 2.06° -0.86° C 5.26 1.85° -0.34° H 6.06 1.72° -0.33° C 3.81 1.74° -0.36° H 3.94 1.47° -0.29° H 3.58 1.75° -0.65°

### E.4 R-GSM

R-GSM is just a subset of GSM8K, with the premise order changes. Here is an example input:

Input: Carmen goes to an auction to win an antique desk. The bids on the desk rise by 50 each time and 3 other people each bid once. She accepts the opening bid of 200 and continues bidding until she wins. Carmen bids after each of the 3 other people and eventually wins. How much money, in dollars, does the desk cost her?

Here The bids on the desk rise by 50 each time and 3 other people each bid once. and She accepts the opening bid of 200 and continues bidding until she wins. are interchangeable.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: - Please discuss the relationship to NoPE (https://arxiv.org/pdf/2305.19466) in your related work
- I find the importance measure that you introduce a little bit weird if you make it independent of positional bias, since you will later calculate attention with positional bias in there, which might change the importance quite a bit. Have you found your measure of document importance, without positional bias, to be monotonic with an importance measure that calculates the attention with positional bias, but putting each document candidate at the same position, right before the currently decoded document for instance?
- I am overall confused by the introduction of bidirectional attention. Given that you calculate importances independently of relative document-position, and you reorder anyway, why do you have to have an attention in the forward direction, instead of simply always putting the document you are decoding at the last position? I might have understood this wrong, the first half of Page 5 is a little confusing to me. When following the concrete example in the proof, there is no bidirectional attention, right?
- Can you comment on the downsides of making keys query dependent (L235)? This seems like you will loose parallelism, at the very least the option to KV-cache.
- Can you comment on your intuition as to why SP and PCW are worse than your approach?
- You say you introduce two other debiasing baselines, permutation and calibration. Is permutation the same as Vanilla (shuffle) in Figure 4? Why are the numbers for it not in any tables?
- Table 2: I don’t fully understand the baselines. IIUC, the GT is either at A or at B, which is why you say 50% random guess baseline, i.e. there is no position C. Why is Vanilla (without qualifier) worse than both Vanilla (GT at A) and (GT at B)? it is not always one of the two, which would lead to some kind of average?
- what do you mean by reasoning “pairs” in 418ff?
soundness: 2
strengths: - very relevant problem
- the recency and primacy biases are interesting.
summary: The authors introduce a method to de-bias a positional dependence for concatenated documents in the context of a LM and prove its supremacy against other de-biasing baselines.

They also compare against vanilla inference, but I cannot tell whether the numbers are actually beneficial.
weaknesses: - Figure 2 is confusing, since the documents change color when you move them (in the right plot). So it looks like the $D_3$ in the first row is the tokens [2,3], but in the second assignment iteration suddenly $D_3$ is [4,5]? It seems inconsistent with the last row, where importance ordering is $D_2 > D_3 > D_1$, but they are not ordered in that way.
- I don’t think you can claim that you pinpointed the causes of positional bias in transformers. Everybody knows what those are.
- Please Correct me if I am wrong here.
The proof in App B shows that the softmax term is invariant under inter-document permutation before the ordering takes place, but Lemma 1 takes the QKV invariance as a given.
How I think Lemma 1 and the proof should go:
Lemma 1: “The PINE algorithm makes $H_{Pine}$  document-position invariant”
Proof: “The algorithm sorts the documents by a document-position invariant metric and concatenates them. Therefore, the resulting concatenation  is invariant. $H_{Pine}$ is only a function of that concatenation, therefore $H_{Pine}$ is invariant. Proof ends”.
- I can’t tell if the numbers are good. The method outperforms other methods of debiasing, but it does not seem to do super well against Vanilla (shuffle) in Figure 4 for instance. Given that you shuffle reward bench, is Vanilla in Table 2 also implicitly Vanilla (shuffle)? It is overall confusing to me, but I admit that this is an opinion.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Figure 2, I understood the last row. 8 is at its original place and (4,5)> (6,7) > (2,3). 
    - Rows 6 and 7 -- (6,7) is at its original place and (2,3) > (4,5)
    - Rows 4 and 5 -- (4,5) should be at their original place, but (6,7) is! Why revise the position of this document to the end when both the documents can be (equally) at small distance from the document under consideration?
- I only understood Figure 2 after going through paragraph 2 in lemma 1. Please consider revising the write up with the document order for computation of each of the rows of the attention.

### Suggestions:
- Move paragraph 2 from Lemma 1 proof from appendix to main paper to explain your method.
- Consider revising the words position invariant to input-position invariant. Please also consider revising the claim that the position bias of LLMs is eliminated (in the title of the paper). From my best understanding, it is mitigated through a reordering and not eliminated.
soundness: 2
strengths: - I like this paper. they propose a bidirectional attention in the input documents to achieve input order independent attention. 
- Their results are solid. For order dependent benchmarks, they do achieve significant gains.
- While the paper is solid, I would like the authors to revise their claims of position invariance to input-position invariance.
summary: Positional bias is a major problem for language models. Several proposals have been made in literature towards mitigating this issue. This paper proposes one method by making the inter-document attention bidirectional. Specifically, they propose their own order such that the effects of order dependence in positional encoding is minimized.
weaknesses: - I tried understanding Figure 2 many times, it was not clear to me. Please see question below. The only sensible explanation is that the attention is simulating the document under question being placed at the end of the sequence, with nearness being provided by importance scores.
- What about the n^2 importance score computations per set of n documents. How to do this when there are multiple documents?
- The position invariance is quite misleading. The correct term I believe would be importance dependent position weighting. i.e. You know the documents far away to the current document under consideration have higher likelihood of error, and hence you place them far away, such that the impact of position is mitigated. 
- Here is a counterexample for position invariance: Let us take a query which involves multi-hop retrieval. The query asks: Retrieve the home pages of all the professors working in the CS department of X university, and list out their topics of interest. Here the retrievals may be all nearly equally important. A full position invariance would require an independence with order whereas that cannot be achieved through this method. What you instead do is to rank documents by importance order and then mitigate positional dependence by placing them in that order of closeness for attention. This is not "invariance". My major concern is with the nomenclature, which claims invariance, and eliminating position bias.
- Lemma 1 may need a revision: sorting by computed importance scores introduces an implicit positional bias. While the function \( f(\text{input order}, \text{content}) \rightarrow \text{output permutation order} \) maintains input order invariance if \( f(\text{io1}, \text{content}) = f(\text{io2}, \text{content}) \) for all input orders \(\text{io1}, \text{io2}\), it imposes a new order-dependent bias in the output, meaning true position invariance is not achieved. Thus, while input-position invariance is maintained, true positional invariance is not achieved, as the sorting process introduces a new, order-dependent bias in the output.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Your method gives position invariance as long as the attention orderings for each document depend only on the content of the documents rather than their positions. Thus, mathematically you can consider any function that maps a list of k documents to a permutation on k-1 documents (k-1 since the document on the diagonal is always in the last position). This class of functions is large and includes special cases like variants of your importance score (e.g., variants regarding the choice of normalization). It seems important to assess the sensitivity of PINE with respect to the choice of ordering function.

2. If position bias is as big a problem for our field as the paper argues, it should be possible to show gains on mainstream benchmarks. Consider the generic position-dependent biases in few-shot prompting, as identified by Zhao et al. in their contextual calibration paper. Could you improve, say, the recency bias of few-shot prompting with your approach? If yes, it would be nice to see this demonstrated on a mainstream LLM benchmark.

3. I am not very familiar with the Qwen series of models. Since your method yields essentially no gains for Llama on RewardBench (as noted above – please correct me if my analysis is mistaken), I wonder if the strong performance on the Qwen models has something to do with the architecture of these models. After looking into the matter, I wonder if the Qwen models’ use of sliding window attention could be the reason why the extra bi-directional attention from your method yields a benefit. What do you think? 

4. The ability to permute documents gives you a way to estimate the variance of vanilla sampling performance. Given the wide disparity between the GT-A and GT-B performances on RewardBench, did you look into randomizing the order of premises on R-GSM? This seems especially important because there are so few samples (only 95) in this data set.
soundness: 3
strengths: 1. The paper is well-written and the graphics are appealing.
2. Position-invariant representations are nice, and you obtain them without k! overhead.
3. The positional bias on RewardBench appears significant and adds credibility to your argument that position bias is an important problem.
4. Strong results for the Qwen models on RewardBench.
5. Strong results for Llama on the reasoning subset of RewardBench.
summary: The paper presents a modification to the causal attention mechanism of decoder-based transformers. The proposed mechanism adds bidirectional attention to each document, in order to make representations position-invariant. New bidirectional attention terms are added in blocks (for the tokens corresponding to each document), which are ordered according to their importance to the document on the diagonal. The proposed importance score is based on a version of the standard attention calculation that ignores positional differences. The paper proves positional invariance and evaluates the empirical performance of the new method on a set of tasks.
weaknesses: 1. You compare your method to other mechanistic methods with different goals: for example, PCW helps extend the context window; NIA gives a speed boost by making the attention computation sub-quadratic. Thus, these are both *approximate* methods. Your method, by contrast, introduces extra overhead compared to standard sampling. For a fair comparison, it is important to look for methods that are more resource-intensive than standard sampling (for example, if your method has 8x overhead, a simple baseline could involve sampling 8 permutations of the documents and running vanilla sampling 8 times, followed by some aggregation scheme like majority voting).

2. I appreciate the analysis shown in Figure 4b) but I would also like to see an analysis of the choice of importance score (see question 1. below).

3. On both RAG-QA and RewardBench, PINE does not seem to improve over vanilla sampling with the Llama models. To see this, note that the average of the GT-A and GT-B scores should give the expected performance for vanilla sampling (with randomized ordering of the two documents -- please correct me if I am wrong). On molecular generation and math reasoning, you switch between evaluating the Llama and Qwen models, which does not inspire confidence; I would like to see the same list of models evaluated for each task. Overall, your selection of tasks seems somewhat contrived. See question 2. below.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: <none>
soundness: 4
strengths: - The approach does not require any extra training, and can be used by any attention mechanism.
- Remarkable improvements on rewardbench evals.
- Comparison with relevant baselines and experiments with multiple models performed.
- Positional invariancy is thoroughly ascertained through empirical and theoretical explanations.
summary: Proposed work introduces PINE, an attention mechanism that computes the attention scores in a positionally invariant manner. The method does not require any training and can be plugged into an transformer inference setup with minimal costs. Improved results in multiple benchmarks show the effectiveness of the approach.
weaknesses: - Regarding,
> Hsieh et al. (2024) assumes that the position bias and real relevance are linear combinations and propose solutions accordingly. Different from them, we aim to eliminate the position bias from the mechanical perspective without any assumption at a reasonable cost.

Although it is claimed that this solution is better motivated than the cited one, a comparison is still required since both papers are solving the same problem and show very good improvements in similar yet different tasks.


- [This](https://openreview.net/pdf?id=gEMLMMG0m9) work debiases positional bias in the attention matrix by averaging attention values of documents at different positions. A mention of why such simple methods to debias are less relevant or a comparison is required.


- Regarding line 288,
> the extra big O computation complexity to obtain hidden states is O(nk log k),

From my understanding, for an end-to-end inference time overhead calculation, this should be multiplied by #layers in the model. And further with #heads if computation across heads is not parallelized. The equation does not capture this.
Hence for lines 518-519,
> we find the wall time of PINE is ~2x and ~8x of the vanilla inference

These numbers should depend on the size of the models being considered. Can you tabulate this for each model separately to give a better picture of overhead time? This is significant as benchmarks tend to use large models as evaluators and an inference time $\propto $ #layers limits the practical applicability of the work.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do different position assigning strategies have a big effect on the downstream performance?
soundness: 3
strengths: 1. This paper proposes a simple yet effective and novel method to mitigate the postion bias problem in LLMs.

2. Extensive experiemnts on Llama-3 and Qwen demostrate the effectiveness of the proposed method.
summary: This work addresses the position bias problem in language models (LMs), where models prioritize information based on its order in the input, affecting performance and reliability across applications. Through a mechanistic analysis, the study identifies causal attention and relative positional encodings as primary contributors to this bias. To mitigate it, the authors introduce Position-INvariant InferencE (PINE), a zero-shot approach that replaces causal attention with bidirectional attention between documents and uses attention values to determine document order. PINE effectively enhances model performance in tasks like QA, molecule generation, and math reasoning, showing notable gains in reasoning benchmarks where it surpasses even state-of-the-art models like GPT-4 in specific evaluations.
weaknesses: 1. The ablation study of the strategy of re-assign positions is required to better discuss the potential of further improving the method.

2. There is no baseline comparison with other calibration based methods.

Meta-review: This paper addresses position bias in transformers in situations where generation is supposed to be conditional on a set of input documents.The proposed approach, dubbed PINE, first re-arranges inputs based on relative pairwise document importance scores as measured via aggregate attention scores, which then results in invariant generation given that re-ordering is carried out prior to generation. This comes at the cost of inference overhead since the re-ordering step requires computing importance. PINE operates during inference only and incurs no extra training cost.

The proposal is evaluated in a number of settings generally showing improvements, however it's a bit unclear how inducing position invariance improves so much the performance in reasoning benchmarks such as in the results shown in tables 2 and 3 for RewardBench. If the assumption is that position-dependent models may miss relevant context due to its position, then that could have been verified by exhaustively checking all possible orderings of input documents (perhaps for a subset of the experiments).

The manuscript has some presentation issues, and the method description in section 3.3. as well as the definition of importance scores are a bit confusing and not clear enough. Another limitation worth highlighting is the fact that PINE is limited in scope to settings where input contexts have well defined boundaries, e.g., it is comprised of a set of documents. In a situation where one has a single large document, position bias could still affect performance since relevant information could occur in parts of the document models tend to ignore. Sub-document re-ordering seems to be required in such a case. Moreover, the evaluation, while extensive in a few aspects, has some limitations that are worth noting. For instance, multi-hop settings where information required to answer queries are spread across multiple documents should have been covered.

In summary, the approach offers a strategy to trade compute for order invariance, and consequently boost performance, in settings where generation is conditional on input context and information needs to be retrieved from a set of documents. Although some limitations remain as discussed in the paragraph above, enough evidence is presented to support the authors's claims and the approach would be of value to the community.

Decision: Accept (Poster)