RoToR: Towards More Reliable Responses for Order-Invariant Inputs
Authors: Soyoung Yoon, Dongha Ahn, Youngwon Lee, Minkyu Jung, HyungJoo Jang, seung-won hwang
Accepted at: acl25

RoToR: Towards More Reliable Responses for Order-Invariant Inputs

Soyoung Yoon1* Dongha Ahn1,2 Youngwon Lee1 Minkyu Jung2 HyungJoo Jang2 Seung-won Hwang1†
1Seoul National University 2Channel Corporation {soyoung.yoon, seungwonh}@snu.ac.kr

Abstract

Mitigating positional bias of language models (LMs) for listwise inputs is a well-known and important problem (e.g., lost-in-the-middle). While zero-shot order-invariant LMs have been proposed to solve this issue, their success on practical listwise problems has been limited. In this work, as a first contribution, we identify and overcome two limitations to make zero-shot invariant LMs more practical: (1) training and inference distribution mismatch arising from modifying positional ID assignments to enforce invariance, and (2) failure to adapt to mixture of order-invariant and sensitive inputs in practical listwise problems. Then, to overcome these issues we propose (1) RoToR, a zero-shot invariant LM for genuinely order-invariant inputs with minimal modifications of positional IDs, and (2) Selective Routing, an adaptive framework that handles both order-invariant and order-sensitive inputs in listwise tasks. On the Lost in the middle (LitM), Knowledge Graph QA (KGQA), and MMLU benchmarks, we show that RoToR with Selective Routing can effectively handle practical listwise input tasks in a zero-shot manner.1

1 Introduction

Language conveys meaning in part through positional information, such as word placement and sentence structure. Given this nature, Language Models (LMs) that learn from human language are trained sensitive to positional information related to the ordering of segments. However, there are some listwise inputs that require neutrality to positional information. For example, for inputs such as sets, tables, databases, or multiple-choice questions, the ordering of the input segments—e.g., rows in a table or elements in an unordered set—require an order-agnostic understanding. We refer to such inputs as “order-invariant inputs,” on which LMs reportedly struggle. For example, in LLM-as-a-judge scenarios, LMs exhibit a preference of up to 75% for the first answer in pairwise inputs (Zheng et al., 2024b), and ranking between LMs can change up to 8 positions in different orderings of multiple choice questions on MMLU (Alzahrani et al., 2024). Such results question the reliability of LMs on order-invariant inputs. Meanwhile, existing methods for enforcing invariance to LMs showed limited effectiveness in real-world tasks, which we hypothesize to arise from the following limitations.

First, training and inference distribution mismatch due to the positional ID re-assignment of zero-shot order-invariant LMs: Fig. 1 illustrates how self-attention is altered in these models. Unlike the original non-invariant model which always assigns position IDs in a causal, ascending manner, order-invariant models either eliminate inter-segment attention, such as PCW (Ratner et al., 2023) in Fig. 1a, or re-assign position IDs as in PINE (Wang et al., 2024) in Fig. 1b, re-ordering segments using pairwise similarity, placing simi-

![Self-attention alteration from order-invariant models. (a) PCW by elimination (b) PINE by re-assignment of position IDs based on query-based pairwise ordering. In contrast, (c) RoToR minimizes the distribution mismatch by global ordering with circular assignment.](page_1012_362_393_246.png)

Figure 1: Self-attention alteration from order-invariant models. (a) PCW by elimination (b) PINE by re-assignment of position IDs based on query-based pairwise ordering. In contrast, (c) RoToR minimizes the distribution mismatch by global ordering with circular assignment.

* Work done during an internship at Channel Corporation.
† Corresponding author.
1 https://github.com/soyoung97/RoToR
lar segments closer to the query. For each query segment, it computes segment-wise query-key attention (for each attention head in each decoder layer) and re-assigns position IDs of segments as keys. This query-dependent segment ordering leads to excessively frequent alterations of positional ID assignments. Frequent re-assignments can also confuse the model and risk collisions which violate the invariance property (e.g., multiple key segments having the same similarity to a query).

To overcome this, we propose a query-agnostic global sorting with circular arrangement for order-invariant positional ID assignment. Ours is named RoToR, inspired by the word rotary to express circular assignment, and also a palindrome, to reflect order invariance. Fig. 1c contrasts with PINE in Fig. 1c, where RoToR only needs a single global ordering (e.g., A->B->O->K->G) with no extra attention computation. The ordering of segments on suffix tokens remains in a fixed order, since it does not rely on their similarity to the query. Finally, we propose three different global sorting algorithms for RoToR, and demonstrate that they consistently outperform previous order-invariant models.

Second, for practical listwise inputs, order-invariant tasks may partially include order-sensitive inputs that require order-specific understanding. For example, the (d) None of the above option in MMLU cannot be reordered. Such a “mixed” nature requires handling each of the cases adaptively, for which we propose a simple Selective Routing method. Selective Routing adapts to a given input by routing between two models, invariant and non-invariant (original), based on the confidence scores of their predictions. Experiments on the MMLU benchmark show that Selective Routing effectively handles datasets with order-invariant and sensitive inputs, and achieves better order robustness while maintaining the original performance.

In summary, our contributions are as follows: 1. Clarifying key challenges to robust understanding of listwise inputs. We pinpoint the distribution mismatch and positional ID assignment complexities that hinder zero-shot order-invariance in LMs, and the need to adaptively handle order-invariant and order-sensitive inputs. 2. A stable, order-invariant solution (RoToR): We propose a query-agnostic global ordering with minimal positional ID modifications, resulting in stable and efficient order-invariance. 3. Adaptive handling of listwise inputs (Selective Routing): We introduce a simple routing method that switches between the original and invariant LMs based on confidence. On MMLU, we show that Selective Routing can adaptively deal with both types of input, leading to better stability. To this end, we aim to develop a model that excels at processing a wide range of listwise inputs reliably and efficiently.

2 Related Works

2.1 Positional bias of LLMs

Problem statement. Recent works on (zero-shot) retrieval augmented generation (RAG) with LLMs have found that the models exhibit unwanted bias on the ordering of the retrieved documents (Chhabra et al., 2024). Widely known as the lost-in-the-middle problem (Liu et al., 2024), many prior studies (Chen et al., 2024; Gupta et al., 2024; Pezeshkpour and Hruschka, 2023; Zhao et al., 2023; Zhou et al., 2024; Wei et al., 2024; Alzahrani et al., 2024; Zheng et al., 2024a) also investigate the impact of positional bias, extending the domain to structured knowledge grounding (SKG) tasks (Zhao et al., 2023; Zhou et al., 2024) and multiple-choice questions (Gupta et al., 2024) where changing the ordering of rows, schemas, or choices greatly degrades performance.

Considerations for decoder-only LMs. While successful approaches are presented to mitigate this issue for encoder-only (Yang et al., 2022) and encoder-decoder (Yen et al., 2024; Cai et al., 2023) models, they leave decoder-only models, which account for the current frontier LLMs, for more consideration. In contrast to transformer encoders that use bidirectional attention which is invariant by nature (Lee et al., 2019), transformer decoders use causal attention to learn causal relation signals, which is not invariant by nature (Haviv et al., 2022a). Therefore, positional bias for decoder-only models is known to stem from both positional encoding and causal attention mask (Yu et al., 2024; Wang et al., 2024) and is harder to mitigate.

2.2 Zero-shot order-invariance for LLMs

Long context modeling. Zero-shot approaches for mitigating positional bias in LLMs were first raised in long-context tasks, with a goal to correctly handle relevant information located in the middle of lengthy inputs2. Nonetheless, these works focus primarily on understanding long texts without losing precision (Li et al., 2023; Zhang et al., 2024a; An et al., 2023; Bai et al., 2024), whereas positional

2github.com/gkamradt/LLMTest_NeedleInHaystack
bias is a more general problem that can occur even on multiple-choices questions with relatively short contexts (Alzahrani et al., 2024). Technically, this line of works modify the attention mechanism by altering the positional encoding to adapt an LLM to longer contexts (Peng et al., 2023; Hsieh et al., 2024; Peyshakhovitch and Lerer, 2023; Chen et al., 2023; Junqing et al., 2023; Xu et al., 2023; Yu et al., 2024; Zhang et al., 2024b). But since they do not modify the causal mask which also contributes to positional bias, order-invariance is not guaranteed in general (Haviv et al., 2022b).

(Zero-shot) order-invariance. Recent line of works focused on achieving order-invariance by mechanistically altering both positional encoding and causal masking. While several works require training (Junqing et al., 2023; Zhu et al., 2023), we focus on zero-shot approaches for practicality, namely PCW (Ratner et al., 2023), Set-Based Prompting (McIlroy-Young et al., 2024), and PINE (Wang et al., 2024), which we explain in detail at Sec. 3.1. Another line of works based on self-consistency try to mitigate positional bias simply by running inference multiple times with different orderings of contexts (Zheng et al., 2024a). However, in principle, this requires evaluating \( n! \) forward passes in total, enforcing Monte Carlo approximations (Tang et al., 2024). More recent work optimizes the number or passes (Lee et al., 2025b) with similar comprehensiveness (Hwang and Chang, 2007), or replaces with contrastive training objective (Lee et al., 2025a). In contrast, our method guarantee invariance with a single forward pass, without requiring any approximations.

3 Methodology

3.1 Baseline: Order-invariant causal LMs
In this section, we briefly overview the existing work on endowing decoder-only models on order-invariance by adjusting attention mechanism, and review their limitations.

Isolated parallel processing Prior works like PCW (Ratner et al., 2023) and Set-Based Prompting (McIlroy-Young et al., 2024) have modified the attention mask and positional ID assignments of the language model to isolate the processing of each segment and apply same positional embeddings are applied across segments, and thus achieve order invariance: However, this design completely prevents one segment from attending to the others, and aggregating the information from different segments is solely handled at suffix and generated tokens, significantly hindering the LM’s cross-segment contextualized understanding of the text. Yang et al. (2023) have argued that this essentially degenerates to mere ensemble of conditioning on each context separately. Such information bottleneck and train-test time discrepancy limits the applicability, more severely as the number of segments is increased.

Bidirectional processing with Q-K similarity A more recent work, PINE (Wang et al., 2024) has addressed these issues through a bidirectional attention mechanism that allows each segment to attend to all other segments. To achieve this within decoder-only models while maintaining order invariance, PINE dynamically modifies positional IDs based on whether a token acts as a query or key in the attention computation.

The key insight is that PINE creates an “illusion” for each query segment: it assigns the query segment the largest positional IDs among all segments, enabling it to attend to all other segments bidirectionally. The ordering of key segments is then determined by their relevance scores computed without positional embeddings (\( \mathrm{Attn}_{\mathrm{NoPos}} \)), ensuring that more relevant segments appear closer to the query.

Consider the example in Fig. 2 with input [T1‘Given’, S1[Apple], S2[Ban”, ana], S3[Orange]], T6‘which one’, . . . T10‘red?’]. The prefix token ‘Given’ (T1) and suffix tokens ‘which one .. red?’ (T6-T10) maintain their original positions and follow standard causal attention. For the segments S1-S3, PINE applies its order-invariant mechanism:

Dynamic positional ID assignment: When a token from segment S2 (e.g., ‘ana’ at T4) acts as a query, PINE: (1) Assigns S2 the highest positional IDs (4-5) among all segments, placing it last. (2) Maintains internal causal order within S2: ‘ban’ gets position 4, ‘ana’ gets position 5. Then, it (3) Reorders other segments (S1, S3) based on their \( \mathrm{Attn}_{\mathrm{NoPos}} \) scores with S2. Conversely, when the same token acts as a key for another query (e.g., from S1), its position depends on the relevance score between S2 and the query segment. If \( \mathrm{Attn}_{\mathrm{NoPos}}(S1, S2) > \mathrm{Attn}_{\mathrm{NoPos}}(S1, S3) \), then S2 is placed closer to S1 than S3.

This dynamic reassignment occurs for every attention computation: each query token sees a different positional arrangement of the key tokens, determined by their relevance scores. Prefix, suffix, and generated tokens do not participate in this re-
Figure 2: Attention mask and positional ID modifications for segment-wise order invariance using example input “Given Apple Banana Orange...”. Each block represents a single token arranged by positional ID assignment. In practice, tokens retain original positions but receive reassigned positional IDs as shown. White areas indicate masked attention. Key difference: PINE requires per-query sorting while RoToR reuses one global ordering across all queries.

ordering and always maintain their standard causal positions. However, when these non-segment tokens act as queries, they still see the segments reordered by their relevance scores.

3.2 RoToR: minimal OOD from positional ID assignments

Challenges with PINE While PINE achieves order-invariance by contextualization across segments, its query-specific ordering scheme introduces (1) significant train-test behavior discrepancy as well as (2) unnecessary complexity and numerical instability, which limits its scalability. During decoding with PINE, position IDs are assigned differently for every query token (each token in the suffix), decoder layer, and attention head, as the query-key attention score \( \text{Att}_{\text{NoPos}} \) determines the position IDs. This complexity introduces excessively frequent alterations on position IDs: As the base LM is trained with fixed positional IDs and causal masks, this causes hidden activations higher risk of out-of-distribution (OOD) for it to process properly. Moreover, ordering segments based on attention is computationally expensive and introduces numerical instability. As computing the attention value of one query segment requires computing the KV attention over every other number of segments, PINE invokes \( \mathcal{O}(n^2) \) cost overhead for each segment for input length \( n \), which is further multiplied by the number of all combinations of layers, heads, and the number of suffix and generated tokens. Also, in practice, calculating attention without RoPE results in a very narrow range of values. bfloat16 numeric type lacks precision to distinguish these values, leading to non-determinism originating from several tied values. The outcome may then depend on the initial ordering.

Motivation & Theoretical Foundation While investigating ways to overcome the limitations of PINE, our central goal is to preserve order invariance while minimizing the complexity of the reassignment of positions. We reason that defining a single global ordering scheme, not necessarily relying on attention scores, and re-using them across all queries can solve the problems stemming from query-dependent ordering. A circular assignment of a global order seems as a practical solution. The idea of using global sorting to achieve ordering invariance has been studied in set/graph ML domain (Murphy et al., 2019a,b), but to the best of our knowledge, using circular assignment of the global ordering, and the application to pre-trained language models, are our novel contributions. As a result, we propose RoToR(Fig. 3), which uses one global ordering that is not a function of the initial ordering of segments (e.g., canonical ordering by lexical sorting) and assigns IDs for tokens in different segments based on circular arrangement.

Global ordering Instead of re-computing the relative order of segments for each query, we reuse a globally shared single ordering, avoiding costly recomputation of numerically unstable attention
Figure 3: Comparing the ordering of 5 segments (S1 - S5) of PINE (Wang et al., 2024; left) and RoToR (Ours; right). PINE sorts segments using aggregated attention scores. In order to be fully ordering-invariant, segment sorting is changed per token in suffix level, causing confusion. In contrast, we define one global sorting of segments and conduct circular assignment between segments. With this, we simply use the global sorting for position id assignment on suffix tokens, without harming invariance.

scores. Moreover, this further reduces the gap between the LLM’s pretrained behavior and test-time behavior, as consistent position IDs are assigned across layers/heads/across suffix tokens. Global ordering allows to preserve the relative placement of segments, further closing the gap induced from introducing order invariance to causal LMs. For example, in Figure 3, due to the global ordering, segments \( S_5 \) and \( S_2 \) are always placed in adjacent positions with RoToR (right side), while it is not satisfied and constantly changed with PINE (left side). We consider three separate global sorting algorithm to be used in RoToR: (1) simple lexicographical sorting which can be obtained with minimal overhead based on tokenized sequence of segments, (2) using a pointwise reranker (Nogueira et al., 2020)\footnote{castorini/monot5-base-msmarco-10k} to score relevancy of each row with respect to the question, or (3) simple frequency-based sorting which normalizes token ids based on the inverse frequency of each token (Details at Appendix Fig. 6). Empirically, we find that using simple lexicographical sorting is sufficient to obtain improvements over PINE.

Circular arrangement To mimic bidirectionality with causal LMs, each segment should be assigned position IDs so that they appear to themselves as being placed at the end of the sequence of segments. To achieve this with a shared global ordering, we employ circular arrangement, each segment taking turns to be placed at the end while their relative ordering is preserved. Given the global ordering, we can construct a single directed graph by combining the front and last parts. Then, we assign orderings for each segment as query by following the path from the graph, starting from the query segment, which is illustrated in Fig. 3. For all suffix and generated tokens, segments are arranged according to the initial front and last part of the global ordering. Compared to PINE where we have to assign different orderings of segments for each suffix and generated tokens, RoToR assign the same positional ID, acting merely the same as the original token. This also accounts for reducing the distributional gap between the original model.

Computational overhead We report only operations executed beyond vanilla self-attention cost \( O(n^2 d) \), where \( n \) is total input length, \( d \) is hidden dimension, and \( k \) is the number of segments. PINE requires two additional operations: (1) computing attention scores without rotary position embeddings (\( O(n^2 d) \)) and (2) sorting \( k \) segments for each query token (\( O(nk \log k) \)), totaling \( O(n^2 d + nk \log k) \) (Wang et al., 2024)\footnote{The PINE paper reports \( O(nk \log k) \) by absorbing the \( O(n^2 d) \) term into baseline; we expose it explicitly for fair comparison.}. In contrast, our lexicographical sorting requires only a single global sort of \( k \) segments (\( O(k \log k) \)), each with length \( O(n) \), achieving \( O(nk \log k) \) and eliminating the expensive \( O(n^2 d) \) term entirely. This can be further optimized to \( O(nk) \) using radix sort.\footnote{https://en.wikipedia.org/wiki/Radix_sort} We empirically validate significantly faster performance than PINE as \( k \) increases (Tab. 4).

3.3 Selective Routing for handling order-sensitive inputs

Since many practical benchmarks such as MMLU involves semi-invariant inputs, we propose a routing mechanism that uses the order-invariant model in conjunction with the standard causal model for further applicability. Our design is partly based on the finding from Wei et al. (2024) that there is
correlation between task difficulty (which is in turn correlated with confidence values) and the model’s sensitivity to ordering. Selective Routing, illustrated in Fig. 4, combines confidence, the model output probability for the generated answer, from two different model versions—the original model and the order-invariant model—on the same input and choose a more confident answer. Both models first produce a maximum probability over possible answer tokens (e.g., A, B, C, D for MMLU) and a corresponding answer choice. We then compare the original model’s maximum probability, plus a bias term \( \alpha \), to the invariant model’s maximum probability. If the original model’s adjusted score is higher, we take its answer; otherwise, the invariant model’s answer is chosen. \( \alpha \) is a hyperparameter that controls how strongly the original model is favored, which was selected as 0.2 according to hyperparameter search on the validation subset (Appendix Sec. E).

4 Experiment setup

4.1 Baselines

Original causal LM with no modifications (Orig.) were compared, which processes text sequentially. Also, we compare RoToR against previous zero-shot order-invariant LMs discussed in Sec. 3.1, namely PCW (Ratner et al., 2023), PINE (Wang et al., 2024), and Set-Based Prompting (McIlroy-Young et al., 2024). We use the LLaMA 3.1 (AI, 2024) 8B-Instruct\footnote{meta-llama/Meta-Llama-3.1-8B-Instruct} 70B-Instruct, Qwen1.5-4B-Chat, Qwen1.5-7B-Chat\footnote{Qwen/Qwen1.5-4B/7B-Chat}, and Qwen1.5-72B-Chat as our backbone model for our experiments. As our method doesn’t need training, a single A6000 GPU was sufficient to run all of the experiments except for the Llama-3.1-70B-Instruct and Qwen1.5-72B-Chat model. We also conduct experiments on a subset of benchmarks (LitM and MMLU) on the runtime latency, perplexity, and collision rate of PINE and RoToR, to further validate our claims on Sec. 3.2.

![Illustration of Selective Routing (Sec. E).](page_120_120_627_180.png)

Figure 4: Illustration of Selective Routing (Sec. E).

4.2 Benchmarks with listwise inputs

We experiment with three benchmarks involving real-world listwise input data. Examples of exact inputs and outputs are provided in Appendix G. All reported scores are rounded to the nearest tenth, except for the standard deviation (rounded to the second decimal place).

Knowledge Graph QA (KGQA) In KGQA tasks, the model takes facts over knowledge graphs represented as (subject, relation, object), and answers the given question based on the given facts. We basically follow the KGQA dataset preprocessing and evaluation setup from Back et al. (2023), which uses Mintaka (Sen et al., 2022) with Wikidata for knowledge source, and use the Exact Match (EM), Accuracy (Acc), and F1 score metric for evaluation. We also use MPNet (Song et al., 2020) as a dense retriever to retrieve top-k facts over each question, and experiment with segment size of 30 and 50. Replication details and example dataset format are at Appendix Sec. C and Fig. 10. Along with measuring the performance of the initial input ordering, we report performance after we shuffle the order of segments with 3 different seeds to see shuffle robustness.

Lost in the middle (LitM) We use the Lost in the Middle (LitM) benchmark (Liu et al., 2024), which draws from 2655 queries in the Natural Questions (NQ) dataset. It provides sets of (10, 20, 30) passages, placing the gold passage at predetermined positions (e.g., 0, 4, 9) and filling the remaining slots with irrelevant passages. Following Liu et al. (2024), the best_subspan_em metric is used. Experiments on LitM found that eliminating the effect of index bias is another important detail for measuring true order robustness: (Appendix Sec. H). Thus, we report experiments with index bias eliminated. The exact prompt and full results including index bias is reported at Appendix Fig. 8 and Sec. A.

MMLU The Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2021) (prompts at Appendix Fig. 11) consists of 57 diverse sub-tasks with a total of 14,015 queries to measure general performance of LMs about the knowledge of the world. Despite its popularity, many works report that performance fluctuates heavily depending on the order of choices (Gupta et al., 2024; Pezeshkpour and Hruschka, 2023; Wei et al., 2024; Alzahrani et al., 2024; Zheng et al., 2024a) and is widely investigated to measure the positional bias of the model. We notice that a lot
<table>
  <tr>
    <th>Total ndoc (segments)</th>
    <th colspan="4">10</th>
    <th colspan="4">20</th>
    <th colspan="4">30</th>
  </tr>
  <tr>
    <th>Gold idx at:</th>
    <th>0</th><th>4</th><th>9</th>
    <th>0</th><th>4</th><th>9</th><th>14</th>
    <th>0</th><th>4</th><th>9</th><th>14</th><th>19</th><th>24</th><th>29</th>
  </tr>
  <tr>
    <th colspan="15">Llama 3-1.8B-Instruct</th>
  </tr>
  <tr>
    <td>Original</td><td>54.7</td><td>53.0</td><td>50.2</td><td>54.8</td><td>52.6</td><td>52.8</td><td>52.4</td><td>51.0</td><td>55.6</td><td>51.5</td><td>52.4</td><td>52.8</td><td>52.1</td><td>52.3</td><td>53.0</td>
  </tr>
  <tr>
    <td>PCW</td><td>12.4</td><td>11.9</td><td>12.2</td><td>3.7</td><td>4.0</td><td>4.0</td><td>4.0</td><td>3.9</td><td>2.3</td><td>1.8</td><td>2.0</td><td>2.0</td><td>2.1</td><td>2.0</td><td>2.0</td>
  </tr>
  <tr>
    <td>Set-Based Prompting</td><td>42.5</td><td>42.5</td><td>42.5</td><td>26.3</td><td>26.3</td><td>26.3</td><td>26.3</td><td>26.3</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td>
  </tr>
  <tr>
    <td>PINE</td><td>58.6</td><td>58.8</td><td>59.0</td><td>56.2</td><td>55.7</td><td>55.5</td><td>55.7</td><td>55.5</td><td>54.2</td><td>54.8</td><td>54.3</td><td>53.7</td><td>54.8</td><td>54.2</td><td>54.0</td>
  </tr>
  <tr>
    <td>RoToR-lexical</td><td>61.4</td><td>61.6</td><td>61.6</td><td>61.4</td><td>59.8</td><td>59.6</td><td>59.6</td><td>59.8</td><td>59.2</td><td>59.5</td><td>59.4</td><td>59.1</td><td>59.0</td><td>59.3</td><td>59.1</td>
  </tr>
  <tr>
    <td>RoToR-reversed lexical</td><td>61.6</td><td>61.8</td><td>61.8</td><td>58.9</td><td>59.3</td><td>58.8</td><td>58.6</td><td>58.7</td><td>57.9</td><td>58.2</td><td>57.9</td><td>57.4</td><td>57.9</td><td>57.6</td><td>57.5</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>61.2</td><td>61.4</td><td>61.2</td><td>60.9</td><td>61.0</td><td>61.2</td><td>61.2</td><td>61.2</td><td>60.9</td><td>60.7</td><td>60.7</td><td>60.7</td><td>60.8</td><td>60.8</td><td>60.7</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>61.0</td><td>61.1</td><td>61.1</td><td>60.4</td><td>60.3</td><td>58.6</td><td>60.2</td><td>60.0</td><td>59.3</td><td>60.4</td><td>59.7</td><td>59.5</td><td>59.5</td><td>59.6</td><td>59.2</td>
  </tr>
  <tr>
    <th colspan="15">Llama 3.1-70B-Instruct</th>
  </tr>
  <tr>
    <td>Original</td><td>66.2</td><td>65.7</td><td>65.7</td><td>65.2</td><td>64.3</td><td>65.0</td><td>66.2</td><td>64.8</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>PINE</td><td>67.9</td><td>67.8</td><td>67.5</td><td>65.9</td><td>65.7</td><td>65.9</td><td>65.8</td><td>65.5</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR</td><td>69.6</td><td>69.5</td><td>69.3</td><td>67.6</td><td>67.8</td><td>67.8</td><td>67.7</td><td>67.9</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>68.9</td><td>69.0</td><td>68.8</td><td>67.5</td><td>67.5</td><td>67.7</td><td>67.5</td><td>67.6</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <th colspan="15">Qwen1.5-4B-Chat</th>
  </tr>
  <tr>
    <td>Original</td><td>61.3</td><td>54.8</td><td>53.1</td><td>59.5</td><td>49.1</td><td>47.9</td><td>45.9</td><td>48.3</td><td>56.8</td><td>45.6</td><td>44.9</td><td>44.6</td><td>45.3</td><td>43.5</td><td>48.3</td>
  </tr>
  <tr>
    <td>PINE</td><td>57.2</td><td>57.4</td><td>57.0</td><td>48.6</td><td>48.2</td><td>48.1</td><td>48.9</td><td>46.4</td><td>45.9</td><td>46.7</td><td>46.6</td><td>46.4</td><td>46.4</td><td>46.3</td><td>46.3</td>
  </tr>
  <tr>
    <td>RoToR</td><td>58.5</td><td>58.4</td><td>58.1</td><td>49.9</td><td>49.7</td><td>49.6</td><td>49.8</td><td>49.9</td><td>44.6</td><td>44.8</td><td>44.7</td><td>44.7</td><td>44.9</td><td>44.8</td><td>44.7</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>58.9</td><td>58.5</td><td>58.7</td><td>52.2</td><td>52.1</td><td>52.1</td><td>52.2</td><td>52.6</td><td>50.6</td><td>50.7</td><td>50.5</td><td>50.6</td><td>50.5</td><td>50.6</td><td>50.4</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>56.7</td><td>56.9</td><td>56.9</td><td>51.9</td><td>51.5</td><td>51.8</td><td>51.6</td><td>52.4</td><td>46.8</td><td>46.7</td><td>46.7</td><td>46.4</td><td>47.0</td><td>46.8</td><td>46.6</td>
  </tr>
  <tr>
    <th colspan="15">Qwen1.5-7B-Chat</th>
  </tr>
  <tr>
    <td>Original</td><td>72.5</td><td>63.3</td><td>62.9</td><td>72.5</td><td>58.5</td><td>56.1</td><td>56.0</td><td>58.2</td><td>73.1</td><td>58.6</td><td>55.8</td><td>53.3</td><td>53.2</td><td>52.5</td><td>57.5</td>
  </tr>
  <tr>
    <td>PINE</td><td>65.4</td><td>65.5</td><td>66.3</td><td>59.1</td><td>59.4</td><td>59.1</td><td>58.6</td><td>59.2</td><td>58.0</td><td>55.3</td><td>55.7</td><td>56.3</td><td>55.1</td><td>55.8</td><td>56.1</td>
  </tr>
  <tr>
    <td>RoToR</td><td>68.6</td><td>68.7</td><td>68.6</td><td>62.6</td><td>62.9</td><td>62.7</td><td>63.0</td><td>62.7</td><td>57.0</td><td>57.3</td><td>59.7</td><td>57.4</td><td>57.3</td><td>62.8</td><td>57.0</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>68.8</td><td>69.4</td><td>69.0</td><td>65.2</td><td>65.5</td><td>65.0</td><td>64.9</td><td>65.0</td><td>62.6</td><td>62.8</td><td>62.9</td><td>62.7</td><td>62.9</td><td>62.8</td><td>62.5</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>68.2</td><td>68.4</td><td>68.4</td><td>62.6</td><td>62.9</td><td>62.8</td><td>62.7</td><td>62.3</td><td>59.5</td><td>59.8</td><td>59.7</td><td>59.6</td><td>59.7</td><td>59.7</td><td>59.7</td>
  </tr>
</table>

Table 1: The best_subspan_em (%) scores on the lost in the middle (LitM) benchmark, with indexing bias removed, across varying numbers of documents (ndoc \( \in \{10, 20, 30\} \)) and models. RoToR shows the best performance across all setups. Experiments on ndoc=30 for the Llama 70B model were unable to report due to resource constraints.

of proportions consist of ordering-sensitive inputs, which showed the effectiveness of adaptively applying Selective Routing. We additionally report the average performance for all possible (4!-1) reorderings.

5 Results & Analysis

We report results for KGQA in Tab. 2, and results for MMLU in Tab. 3. Results for LitM are in Tab. 1, with a visualization in Appendix Fig. 5. We use lexical sorting for RoToR unless stated otherwise.

Effectiveness of RoToR We observe that shuffling input segments leads to non-trivial performance degradations in the original model, which exhibits a statistically significant performance drop on our experimented dataset (two-tailed t-test, p < 0.05, Appendix I). In contrast, our proposed RoToR model does not show a statistically significant difference in performance before and after shuffling, indicating that it is more robust against such perturbations. On LitM (Tab. 1), we notice PCW and Set-Based Prompting has impractical performance, with PINE degrading heavily as number of documents (\( k \)) increases, while RoToR is less affected. On KGQA (Tab. 2), we show RoToR outperform PINE with lower standard deviation across shuffled segments, consistent with different model architectures.

Improvements from PINE Experiments against comparing RoToR with PINE (Tab. 4) we analyze FLOPS: 8 RoToR consistently reduces the floating point operations overhead across segment counts and different model backbones compared to PINE. This is because RoToR does not require computing additional attention scores: it only performs tensor operations for circular arrangement. In contrast, PINE requires more cost due to attention-based reassignment. Runtime, scalability: Actual inference times (Appendix Sec. F) find that RoToR outperforms PINE substantially, with efficiency gains increasing alongside \( n \). For instance, on LitM (30 docs), RoToR achieves a 43% reduction in total runtime. Practical scalability with increasing \( k \) is critical, but we find that previous order-invariant LMs struggle handling larger \( k \) (on KGQA and LitM). In contrast, RoToR shows better perfor-

8We used the FlopsProfiler of the DeepSpeed library to measure FLOPS.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Llama-3.1-SB-Instr.</th>
    <th colspan="3">Llama-3.1-70B-Instr.</th>
    <th colspan="3">Qwen1.5-4B-Chat</th>
    <th colspan="3">Qwen1.5-7B-Chat</th>
    <th colspan="3">Qwen1.5-72B-Chat</th>
  </tr>
  <tr>
    <th>Acc.</th><th>EM</th><th>F1</th>
    <th>Acc.</th><th>EM</th><th>F1</th>
    <th>Acc.</th><th>EM</th><th>F1</th>
    <th>Acc.</th><th>EM</th><th>F1</th>
    <th>Acc.</th><th>EM</th><th>F1</th>
  </tr>
  <tr>
    <td colspan="18"><b>N = 30</b></td>
  </tr>
  <tr>
    <th colspan="18">Initial, no shuffling of segments</th>
  </tr>
  <tr>
    <td>Original</td><td>50.2</td><td>44.0</td><td>51.9</td><td>61.6</td><td>57.7</td><td>63.6</td><td>30.7</td><td>27.9</td><td>34.9</td><td>31.5</td><td>27.8</td><td>35.4</td><td>41.4</td><td>37.7</td><td>43.7</td>
  </tr>
  <tr>
    <td>PINE</td><td>51.5</td><td>45.0</td><td>52.6</td><td>63.1</td><td>58.7</td><td>64.8</td><td>31.6</td><td>28.7</td><td>35.6</td><td>32.3</td><td>28.8</td><td>36.4</td><td>46.7</td><td>42.9</td><td>49.0</td>
  </tr>
  <tr>
    <td>RoToR</td><td>53.1</td><td>46.5</td><td>54.1</td><td>63.6</td><td>59.1</td><td>65.2</td><td>32.0</td><td>29.0</td><td>35.7</td><td>34.3</td><td>29.8</td><td>37.7</td><td>47.5</td><td>43.2</td><td>49.2</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>51.6</td><td>45.0</td><td>52.5</td><td>–</td><td>32.3</td><td>29.1</td><td>36.2</td><td>32.9</td><td>28.4</td><td>36.3</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>52.6</td><td>46.1</td><td>53.7</td><td>–</td><td>32.3</td><td>29.2</td><td>36.0</td><td>33.7</td><td>29.5</td><td>37.2</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <th colspan="18">After shuffling segments, averaged over 3 seeds</th>
  </tr>
  <tr>
    <td>Original</td><td>49.5</td><td>43.3</td><td>51.0</td><td>62.1</td><td>57.8</td><td>64.0</td><td>30.1</td><td>27.5</td><td>34.7</td><td>31.4</td><td>27.3</td><td>35.0</td><td>41.0</td><td>37.6</td><td>43.6</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.07 / 0.14 / 0.17</td><td>0.37 / 0.40 / 0.27</td><td>0.41 / 0.34 / 0.43</td><td>0.26 / 0.28 / 0.29</td><td>0.75 / 0.40 / 0.33</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>PINE</td><td>51.8</td><td>45.2</td><td>52.8</td><td>63.3</td><td>58.8</td><td>64.9</td><td>31.5</td><td>28.7</td><td>35.6</td><td>32.3</td><td>28.8</td><td>35.7</td><td>46.9</td><td>43.3</td><td>49.2</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.05 / 0.07 / 0.16</td><td>0.13 / 0.04 / 0.10</td><td>0.20 / 0.18 / 0.13</td><td>0.17 / 0.20 / 0.13</td><td>0.18 / 0.20 / 0.20</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR</td><td>52.8</td><td>46.2</td><td>53.8</td><td>63.5</td><td>59.1</td><td>65.3</td><td>31.8</td><td>28.8</td><td>35.5</td><td>34.2</td><td>29.9</td><td>37.7</td><td>47.4</td><td>43.1</td><td>49.1</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.05 / 0.05 / 0.02</td><td>0.11 / 0.07 / 0.08</td><td>0.05 / 0.02 / 0.09</td><td>0.09 / 0.07 / 0.06</td><td>0.06 / 0.04 / 0.07</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>51.6</td><td>45.0</td><td>52.6</td><td>–</td><td>32.4</td><td>29.2</td><td>36.3</td><td>33.0</td><td>28.8</td><td>36.5</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.12 / 0.06 / 0.10</td><td>–</td><td>0.04 / 0.02 / 0.13</td><td>0.12 / 0.09 / 0.07</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>52.5</td><td>45.9</td><td>53.5</td><td>–</td><td>32.3</td><td>29.3</td><td>36.0</td><td>33.8</td><td>29.6</td><td>37.4</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.10 / 0.15 / 0.11</td><td>–</td><td>0.13 / 0.16 / 0.09</td><td>0.04 / 0.00 / 0.09</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <th colspan="18">N = 50</th>
  </tr>
  <tr>
    <th colspan="18">Initial, no shuffling of segments</th>
  </tr>
  <tr>
    <td>Original</td><td>50.0</td><td>44.0</td><td>51.7</td><td>62.6</td><td>58.5</td><td>64.5</td><td>31.6</td><td>28.6</td><td>35.8</td><td>31.7</td><td>28.0</td><td>35.7</td><td>42.1</td><td>38.7</td><td>44.5</td>
  </tr>
  <tr>
    <td>PINE</td><td>51.6</td><td>45.1</td><td>52.6</td><td>64.1</td><td>59.8</td><td>65.8</td><td>31.6</td><td>28.8</td><td>35.3</td><td>32.0</td><td>28.5</td><td>35.9</td><td>48.0</td><td>44.1</td><td>49.9</td>
  </tr>
  <tr>
    <td>RoToR</td><td>52.9</td><td>46.0</td><td>53.6</td><td>64.6</td><td>60.0</td><td>66.2</td><td>32.7</td><td>29.6</td><td>36.2</td><td>34.3</td><td>30.1</td><td>38.0</td><td>48.4</td><td>44.3</td><td>50.3</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>52.4</td><td>45.4</td><td>52.8</td><td>–</td><td>32.3</td><td>29.3</td><td>35.9</td><td>32.9</td><td>28.9</td><td>36.6</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>53.1</td><td>46.4</td><td>53.7</td><td>–</td><td>32.3</td><td>29.2</td><td>36.1</td><td>33.5</td><td>29.5</td><td>37.2</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <th colspan="18">After shuffling segments, averaged over 3 seeds</th>
  </tr>
  <tr>
    <td>Original</td><td>49.7</td><td>43.5</td><td>51.0</td><td>62.8</td><td>58.5</td><td>64.5</td><td>30.3</td><td>27.6</td><td>35.0</td><td>31.6</td><td>27.9</td><td>35.5</td><td>42.1</td><td>38.9</td><td>44.7</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.34 / 0.28 / 0.46</td><td>0.29 / 0.28 / 0.05</td><td>0.26 / 0.24 / 0.35</td><td>0.40 / 0.56 / 0.42</td><td>0.30 / 0.40 / 0.35</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>PINE</td><td>51.8</td><td>45.3</td><td>52.7</td><td>64.3</td><td>59.8</td><td>65.9</td><td>31.5</td><td>28.7</td><td>35.3</td><td>31.7</td><td>28.2</td><td>35.7</td><td>48.0</td><td>44.3</td><td>50.3</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.15 / 0.16 / 0.19</td><td>0.16 / 0.15 / 0.14</td><td>0.17 / 0.20 / 0.21</td><td>0.18 / 0.16 / 0.14</td><td>0.02 / 0.04 / 0.05</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR</td><td>52.1</td><td>45.9</td><td>53.5</td><td>64.5</td><td>60.0</td><td>66.1</td><td>32.5</td><td>29.6</td><td>36.1</td><td>34.2</td><td>30.1</td><td>38.0</td><td>48.3</td><td>44.3</td><td>50.3</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.07 / 0.08 / 0.08</td><td>0.02 / 0.02 / 0.01</td><td>0.11 / 0.09 / 0.09</td><td>0.08 / 0.09 / 0.08</td><td>0.05 / 0.09 / 0.05</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td><td>52.2</td><td>45.2</td><td>52.8</td><td>–</td><td>32.3</td><td>29.4</td><td>35.9</td><td>32.8</td><td>28.8</td><td>36.5</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.16 / 0.18 / 0.18</td><td>–</td><td>0.16 / 0.13 / 0.07</td><td>0.16 / 0.09 / 0.07</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td><td>53.1</td><td>46.4</td><td>53.7</td><td>–</td><td>32.4</td><td>29.3</td><td>36.1</td><td>33.7</td><td>29.6</td><td>37.4</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
  <tr>
    <td>→ stddev. (±)</td><td>0.02 / 0.07 / 0.03</td><td>–</td><td>0.09 / 0.04 / 0.06</td><td>0.04 / 0.16 / 0.22</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td>
  </tr>
</table>

Table 2: Mintaka (KGQA) results, with Initial and After-shuffle settings, across different model parameter size and backbones. N refers to the number of top-k segments per query. Rows with “→ stddev.” report standard deviation over 3 seeds.

mance with improved efficiency and robustness.

Perplexity: Lower generation perplexity indicates input representations are closer to in-distribution. On the same LitM dataset, RoToR’s reduced perplexity implies its positional ID assignment effectively mitigates out-of-distribution effects. Collision Rate: PINE’s similarity-based ordering often collides: on average, only 17.3 of 30 similarity values are unique, causing 42% of the segments to be indistinguishable and thus breaking invariance. In contrast, RoToR with lexical sorting only collides if the segment texts are identical. On LitM, this yields zero collisions, preserving full invariance.

Selective Routing MMLU (Tab. 3) is a representative of a task that involves not only order-invariant, but also order-sensitive (e.g., "None of the above"), inputs. Therefore, single use of order-invariant models does not always outperform the original model, limiting applicability of order-invariant models to practical listwise tasks, i.e., we observe significant performance drops for Set-based Prompting in MMLU, falling short of half the performance of the original model on initial ordering. However, using RoToR with Selective Routing to handle order-sensitive inputs outperforms, or is at least competitive as the original model in all possible orderings of candidate choices. Selective Routing improves the generalizability and extends the applicability on practical listwise tasks by adaptively handling order-sensitive inputs. The RoToR + Selective Routing (Oracle) performance on Tab. 3 was evaluated using a relaxed accuracy metric based on the union of predictions from the original and the invariant (RoToR-lexical) model. This improves significantly, which highlights the potential of Selective Routing for further accuracy
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Llama-3.1-8B-Instruct</th>
    <th colspan="3">Qwen1.5-4B-Chat</th>
    <th colspan="3">Qwen1.5-7B-Chat</th>
  </tr>
  <tr>
    <th>Init.</th>
    <th>Rev.</th>
    <th>Avg.</th>
    <th>Init.</th>
    <th>Rev.</th>
    <th>Avg.</th>
    <th>Init.</th>
    <th>Rev.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Orig.</td>
    <td>68.3</td>
    <td>64.8</td>
    <td>65.5 ± 1.0</td>
    <td>53.6</td>
    <td>51.9</td>
    <td>52.6 ± 0.6</td>
    <td>60.1</td>
    <td>56.6</td>
    <td>58.6 ± 0.9</td>
  </tr>
  <tr>
    <td>PCW</td>
    <td>57.0</td>
    <td>55.1</td>
    <td>56.1 ± 1.1</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>Set-Based Prompting</td>
    <td>31.1</td>
    <td>33.0</td>
    <td>31.6 ± 0.8</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>PINE</td>
    <td>64.8</td>
    <td>63.3</td>
    <td>63.6 ± 0.7</td>
    <td>50.5</td>
    <td>49.3</td>
    <td>49.4 ± 0.5</td>
    <td>57.0</td>
    <td>54.4</td>
    <td>55.8 ± 0.9</td>
  </tr>
  <tr>
    <td>RoToR</td>
    <td>63.2</td>
    <td>62.6</td>
    <td>62.8 ± 0.7</td>
    <td>49.6</td>
    <td>47.7</td>
    <td>48.3 ± 0.7</td>
    <td>56.5</td>
    <td>55.8</td>
    <td>56.2 ± 0.6</td>
  </tr>
  <tr>
    <td>→ + S.R.</td>
    <td><b>68.5</b></td>
    <td>65.1</td>
    <td>65.7 ± 0.9</td>
    <td>53.7</td>
    <td>51.8</td>
    <td><b>52.6 ± 0.6</b></td>
    <td><b>60.1</b></td>
    <td><b>57.4</b></td>
    <td><b>58.8 ± 0.7</b></td>
  </tr>
  <tr>
    <td>RoToR - MonoT5</td>
    <td>64.2</td>
    <td>62.9</td>
    <td>63.5 ± 0.5</td>
    <td>49.7</td>
    <td>47.6</td>
    <td>48.7 ± 0.7</td>
    <td>56.2</td>
    <td>54.4</td>
    <td>55.5 ± 0.7</td>
  </tr>
  <tr>
    <td>→ + S.R.</td>
    <td>68.4</td>
    <td>65.2</td>
    <td>65.8 ± 0.9</td>
    <td><b>53.8</b></td>
    <td>51.9</td>
    <td><b>52.6 ± 0.6</b></td>
    <td><b>60.1</b></td>
    <td><b>57.3</b></td>
    <td><b>58.7 ± 0.8</b></td>
  </tr>
  <tr>
    <td>RoToR - Freq.</td>
    <td>64.3</td>
    <td>63.6</td>
    <td>63.8 ± 0.6</td>
    <td>49.9</td>
    <td>47.6</td>
    <td>48.7 ± 0.5</td>
    <td>56.4</td>
    <td>54.7</td>
    <td>55.7 ± 0.7</td>
  </tr>
  <tr>
    <td>→ + S.R.</td>
    <td><b>68.5</b></td>
    <td><b>65.3</b></td>
    <td><b>65.8 ± 0.8</b></td>
    <td>53.7</td>
    <td><b>52.3</b></td>
    <td><b>52.6 ± 0.6</b></td>
    <td>60.0</td>
    <td>57.3</td>
    <td>58.6 ± 0.8</td>
  </tr>
  <tr>
    <td>RoToR + S.R. (Oracle)</td>
    <td>75.0</td>
    <td>71.9</td>
    <td>72.7 ± 1.0</td>
    <td>61.8</td>
    <td>60.1</td>
    <td>61.1 ± 1.0</td>
    <td>68.1</td>
    <td>66.2</td>
    <td>67.2 ± 0.7</td>
  </tr>
</table>

Table 3: Improving applicability to general listwise tasks (MMLU, N=4) with Selective Routing (S.R.), which includes both order-invariant and order-sensitive examples. Init./Rev. refer to original/reversed orderings. Avg. is the average selection ratio across all (4!-1) re-orderings with standard deviation. S.R (Oracle) represents the upper bound with perfect routing accuracy. RoToR with Selective Routing shows improved performance and stability across input re-orderings.

<table>
  <tr>
    <th>Model</th>
    <th>Benchmark</th>
    <th>PINE</th>
    <th>RoToR</th>
    <th>Reduction</th>
  </tr>
  <tr>
    <th colspan="5">(a) Overhead FLOPs, relative to original model</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>MMLU, \( N = 4 \)</td>
    <td>0.59×</td>
    <td>0.55×</td>
    <td>7.6%</td>
  </tr>
  <tr>
    <td></td>
    <td>LitM, \( N = 10 \)</td>
    <td>7.07×</td>
    <td>4.81×</td>
    <td>31.9%</td>
  </tr>
  <tr>
    <td></td>
    <td>LitM, \( N = 30 \)</td>
    <td>22.43×</td>
    <td>15.05×</td>
    <td>32.9%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>KGQA, \( N = 30 \)</td>
    <td>1.27×</td>
    <td>0.94×</td>
    <td>26.0%</td>
  </tr>
  <tr>
    <td></td>
    <td>KGQA, \( N = 50 \)</td>
    <td>1.82×</td>
    <td>1.29×</td>
    <td>29.0%</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B-Chat</td>
    <td>KGQA, \( N = 30 \)</td>
    <td>0.45×</td>
    <td>0.01×</td>
    <td>98.0%</td>
  </tr>
  <tr>
    <td></td>
    <td>KGQA, \( N = 50 \)</td>
    <td>0.58×</td>
    <td>0.03×</td>
    <td>94.8%</td>
  </tr>
  <tr>
    <th colspan="5">(b) End-to-end latency (s)</th>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>LitM, \( N = 10 \)</td>
    <td>57.352</td>
    <td>44.219</td>
    <td>22.9%</td>
  </tr>
  <tr>
    <td></td>
    <td>LitM, \( N = 20 \)</td>
    <td>87.091</td>
    <td>58.680</td>
    <td>32.6%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>MMLU, \( N = 4 \)</td>
    <td>7.371</td>
    <td>6.608</td>
    <td>10.4%</td>
  </tr>
  <tr>
    <td></td>
    <td>LitM, \( N = 10 \)</td>
    <td>18.551</td>
    <td>14.264</td>
    <td>23.1%</td>
  </tr>
  <tr>
    <td></td>
    <td>LitM, \( N = 30 \)</td>
    <td>41.664</td>
    <td>23.569</td>
    <td>43.4%</td>
  </tr>
  <tr>
    <th colspan="5">(c) Perplexity & Collision rate, (on LitM)</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>Perplexity (\( N = 20 \))</td>
    <td>6.91</td>
    <td>6.65</td>
    <td>–</td>
  </tr>
  <tr>
    <td></td>
    <td>Collision rate (\( N = 30 \))</td>
    <td>42.3%</td>
    <td>0 (None)</td>
    <td>–</td>
  </tr>
</table>

Table 4: Unified efficiency comparison of RoToR vs. PINE, reporting (a) Additional FLOPS, (b) Latency, and (c) Perplexity & Collision rate. Columns list each metric for PINE and RoToR, and the relative reduction. Yellow rows separate sub-sections.

gains through optimizing design choices on routing methods, which we plan to explore in future work. Additional analysis on the selection ratio of Selective Routing is reported at Appendix Section L.

Impact of global ordering algorithm While most of our experiments focus on the simplest lexical sorting method, RoToR supports any global sorting approach. To demonstrate this flexibility, we report experiments with various global sorting strategies, including reversed lexical sorting, MonoT5-based reranking, and token frequency-based sorting. Lexical sort is presented as a baseline (lower bound) - a simple algorithm ensuring global sorting. Our experiments on Tab. 1 show that any type of global sorting, with the use of circular assignment is superior than PINE, which relies on pairwise attention arrangements.

Extension to LLMs, other scenarios Experiments on Llama-3.1-70B-Instruct and Qwen-1.5-72B-Chat for LitM and KGQA show consistent and significant improvements over both the original implementation and the PINE baseline, demonstrating RoToR’s generalizability to larger-scale LLMs. We further evaluate robustness on longer-context inputs (LongBench-2WikiMultiHopQA (Bai et al., 2024) in Appendix Section J) and different task templates (KGQA template swap in Appendix Section K). Results indicate that our method retains benefits across longer input scenarios and diverse task templates, including those without explicit input format requirements.

6 Conclusion

Our work addresses order-invariance in listwise inputs by identifying core issues in distribution mismatch and adaptive handling of mixed inputs. Our proposed RoToR by modifying self-attention by global sorting and circular arrangement provides a stable zero-shot order-invariant solution that reduces the complexity of positional ID modification, while Selective Routing adaptively routes between invariant and sensitive LMs to handle real-world scenarios. Together, these methods demonstrate improved performance and reliability on LitM, KGQA, and MMLU benchmarks.
7 Limitations

Our method can utilize any kind of deterministic sorting algorithm, but we have only experimented with limited global sorting algorithms due to time and resource constraints. We plan to investigate potentially better sorting algorithms in the future. Also, current ordering-invariant models are limited to inputs given as prefix + parallel + suffix. It would be beneficial to support more complex structures, such as ability to process multiple order-invariant contexts interleaved with serial text.

8 Acknowledgments

We thank the Channel Corporation for providing GPU resources to run the experiments, and their AI team for providing helpful feedback. We thank the members of LDII Lab and Jinwoo Kim for their constructive comments and the anonymous reviewers for their valuable suggestions. We are also grateful to our former intern, Yezun Chung (KAIST CS), for assisting with experiments on early versions of the Selective Routing approach.

This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS-2021-II212068, Artificial Intelligence Innovation Hub) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)]

References

Meta AI. 2024. Build the future of ai with meta llama 3.

Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushtaykeh, Faisal Mirza, Noof Alotaibi, Nora Altwairesh, Areeb Alowisheq, Ma Saiful Bari, and Haidar Khan. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. Preprint, arXiv:2402.01781.

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. Preprint, arXiv:2307.11088.

Jinheon Baek, Soyeong Jeong, Minki Kang, Jong C. Park, and Sung Ju Hwang. 2023. Knowledge-augmented language model verification. Preprint, arXiv:2310.12836.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. Preprint, arXiv:2308.14508.

Tianle Cai, Kaixuan Huang, Jason D. Lee, and Mengdi Wang. 2023. Scaling in-context demonstrations with structured attention. In Workshop on Efficient Systems for Foundation Models @ ICML2023.

Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939.

Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. 2023. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455.

Anshuman Chhabra, Hadi Askari, and Prasant Mohapatra. 2024. Revisiting zero-shot abstractive summarization in the era of large language models from the perspective of position bias. Preprint, arXiv:2401.01989.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing answer order can decrease mmlu accuracy. Preprint, arXiv:2406.19470.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022a. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022.

Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022b. Transformer language models without positional encodings still learn positional information. Preprint, arXiv:2203.16634.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.

Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406.16008.

Seung-won Hwang and Kevin Chen-chuan Chang. 2007. Optimizing top-k queries for middleware access: A unified cost-based approach. ACM Trans. Database Syst., 32(1):5–es.

He Junqing, Pan Kunhao, Dong Xiaoqun, Song Zhuoyang, Liu Yibo, Liang Yuxin, Wang Hao, Sun Qiangou, Zhang Songxin, Xie Zejian, et al. 2023. Never lost in the middle: Improving large language models via attention strengthening question answering. arXiv preprint arXiv:2311.09198.

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Koziorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In ICML.

Youngwon Lee, Seung won Hwang, Daniel Campos, Filip Graliński, Zhewei Yao, and Yuxiong He. 2025a. Cord: Balancing consistency and rank distillation for robust retrieval-augmented generation. NAACL.

Youngwon Lee, Seung won Hwang, Daniel Campos, Filip Graliński, Zhewei Yao, and Yuxiong He. 2025b. Inference scaling for bridging retrieval and augmented generation. NAACL.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Reid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang, and Cynthia Dwork. 2024. Set-based prompting: Provably solving the language model order dependency problem. Preprint, arXiv:2406.06581.

Ryan L. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. 2019a. Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs. Preprint, arXiv:1811.01900.

Ryan L. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. 2019b. Relational pooling for graph representations. Preprint, arXiv:1903.02541.

Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. Preprint, arXiv:2003.06713.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071.

Alexander Peyskahovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427.

Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483.

Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6383–6402, Toronto, Canada. Association for Computational Linguistics.

Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1604–1619, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Preprint, arXiv:2004.09297.

Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Farhan Ture. 2024. Found in the middle: Permutation self-consistency improves listwise ranking in large language models. Preprint, arXiv:2310.07712.

Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. 2024. Eliminating position bias of language models: A mechanistic approach. Preprint, arXiv:2407.01100.

Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. Preprint, arXiv:2406.03009.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025.

Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for table-text encoding. Preprint, arXiv:2203.00274.

Kejuan Yang, Xiao Liu, Kaiwen Men, Aohan Zeng, Yuxiao Dong, and Jie Tang. 2023. Revisiting parallel context windows: A frustratingly simple alternative and chain-of-thought deterioration. Preprint, arXiv:2305.15262.

Howard Yen, Tianyu Gao, and Danqi Chen. 2024. Long-context language modeling with parallel context encoding. Preprint, arXiv:2402.16617.

Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. 2024. Mitigate position bias in large language models via scaling a single dimension. arXiv preprint arXiv:2406.02536.

Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi yang, Junhao Liu, Longze Chen, Run Luo, and Min Yang. 2024a. Marathon: A race through the realm of long context with large language models. Preprint, arXiv:2312.09542.

Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024b. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797.

Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. 2023. Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations. Preprint, arXiv:2306.14321.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024a. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024b. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.

Wei Zhou, Mohsen Mesgar, Heike Adel, and Annemarie Friedrich. 2024. Freb-tqa: A fine-grained robustness evaluation benchmark for table question answering. Preprint, arXiv:2404.18585.

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgem: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631.
Appendix

A  Full results on the Lost in the Middle Benchmark

![Three line plots showing performance with ndoc=10, 20, and 30 for different methods, with and without index bias.](page_186_154_1077_312.png)

(a) Results with index bias (indexed by numbers). (Example input at Appendix Fig. 7)

![Three line plots showing performance with ndoc=10, 20, and 30 for different methods, with and without index bias.](page_186_495_1077_312.png)

(b) Results without index bias (indexed by title) (Example input at Appendix Fig. 8)

Figure 5: Results on the Lost-in-the-middle benchmark. Visualization of the best_subspan_em results at Appendix Tab. 1. RoTOR (dark red, red, yellow) generally performs the best regardless of the position of the gold index, with less fluctuations when we remove index bias. Ours is RoTOR with lexical sort, and ours-reversed is the one with the reversed lexical ordering. For brevity, only the performance of RoTOR with reranking sort (MonoT5) is annotated as numbers, and the performance of PCW and Set-Based Prompting are reported only at the Table (Appendix Tab. 1) due to its low performance.

Impact of removing index bias on LitM  Tab.5 presents the full results on the Lost in the Middle (LitM) benchmark, comparing scenarios where indexing bias is present versus removed. Fig.5 provides a visual representation of these results.

As shown in Appendix Tab.5, invariant LMs exhibit stable performance regardless of the gold index, especially when index bias is removed (as described in Sec.4.2; see also Appendix Fig.5b). However, when index bias is present, performance fluctuations are observed (Appendix Fig.5a). Notably, RoTOR achieves the highest performance across all setups, demonstrating its effectiveness in mitigating positional bias in a zero-shot setting while maintaining overall performance.

These findings suggest that index bias acts as an implicit source of additional positional bias and that invariant LMs benefit significantly from its removal.

B  Illustration of the global sorting method

We show the three different global sorting algorithms presented in our paper at Fig. 6.

C  Details about preprocessing and evaluation of datasets

C.1  General.

All inferences were done with a single NVIDIA RTX A6000 48GB GPU. Note that all of the baseline models including our method can be applied directly in a zero-shot, training-free manner. For reproducibility, we fix the seed and disabled random sampling (i.e., used greedy decoding), set the maximum number of new generated tokens to 500, and set the pad_token_id to the same value as the eos_token_id
<table>
  <tr>
    <th>Total ndoc (segments)</th>
    <th colspan="4">10</th>
    <th colspan="4">20</th>
    <th colspan="6">30</th>
  </tr>
  <tr>
    <th>Gold idx at:</th>
    <th>0</th><th>4</th><th>9</th><th>avg.</th>
    <th>0</th><th>4</th><th>9</th><th>14</th><th>19</th><th>avg.</th>
    <th>0</th><th>4</th><th>9</th><th>14</th><th>19</th><th>24</th><th>29</th><th>avg.</th>
  </tr>
  <tr>
    <th colspan="18">Indexing bias present</th>
  </tr>
  <tr>
    <td>Original</td><td>52.8</td><td>51.1</td><td>46.7</td><td>50.2</td><td>55.0</td><td>50.7</td><td>50.5</td><td>50.2</td><td>47.5</td><td>50.7</td><td>56.1</td><td>50.8</td><td>51.5</td><td>51.1</td><td>50.8</td><td>50.5</td><td>48.9</td><td>51.4</td>
  </tr>
  <tr>
    <td>PCW</td><td>12.0</td><td>11.9</td><td>12.1</td><td>12.0</td><td>3.4</td><td>3.7</td><td>3.8</td><td>3.9</td><td>3.6</td><td>5.1</td><td>2.1</td><td>2.1</td><td>2.0</td><td>1.9</td><td>2.0</td><td>2.2</td><td>2.0</td><td>2.0</td>
  </tr>
  <tr>
    <td>Set-Based Prompting</td><td>40.8</td><td>40.7</td><td>40.8</td><td>40.8</td><td>25.6</td><td>25.8</td><td>25.7</td><td>25.5</td><td>25.3</td><td>25.6</td><td>15.8</td><td>15.9</td><td>16.1</td><td>16.0</td><td>16.1</td><td>15.7</td><td>15.8</td><td>15.9</td>
  </tr>
  <tr>
    <td>PINE</td><td>59.2</td><td>56.8</td><td>57.7</td><td>57.9</td><td>56.2</td><td>55.1</td><td>54.5</td><td>54.2</td><td>55.3</td><td>55.5</td><td>56.1</td><td>53.6</td><td>53.3</td><td>53.9</td><td>53.2</td><td>53.7</td><td>54.5</td><td>54.0</td>
  </tr>
  <tr>
    <td>RoToR-lexical</td><td><b>63.6</b></td><td>60.5</td><td>59.1</td><td>61.1</td><td>52.6</td><td>58.6</td><td>57.8</td><td>56.8</td><td>58.6</td><td>57.6</td><td><b>64.6</b></td><td><b>58.9</b></td><td>56.9</td><td>56.2</td><td>57.1</td><td>56.2</td><td>57.1</td><td>58.1</td>
  </tr>
  <tr>
    <td>RoToR-reversed lexical</td><td>61.5</td><td>60.8</td><td>60.6</td><td>61.0</td><td>60.8</td><td>58.6</td><td>59.7</td><td><b>60.5</b></td><td>59.3</td><td>60.0</td><td>61.1</td><td>57.5</td><td>58.2</td><td><b>58.6</b></td><td><b>58.3</b></td><td><b>59.3</b></td><td><b>59.2</b></td><td>58.9</td>
  </tr>
  <tr>
    <td>RoToR-reranking</td><td>61.4</td><td><b>61.4</b></td><td><b>61.7</b></td><td><b>61.5</b></td><td><b>62.3</b></td><td><b>59.2</b></td><td>59.1</td><td>59.4</td><td><b>59.8</b></td><td><b>60.2</b></td><td>62.2</td><td>58.6</td><td><b>58.4</b></td><td><b>58.6</b></td><td><b>58.3</b></td><td><b>58.5</b></td><td><b>59.8</b></td><td><b>59.2</b></td>
  </tr>
  <tr>
    <td>RoToR-freq</td><td>62.8</td><td>61.1</td><td>59.5</td><td>61.1</td><td>62.9</td><td>58.8</td><td>56.7</td><td>57.4</td><td>58.0</td><td>59.1</td><td>61.7</td><td>58.2</td><td>56.9</td><td>56.1</td><td>56.4</td><td>55.4</td><td>56.8</td><td>57.4</td>
  </tr>
  <tr>
    <th colspan="18">Indexing bias removed (main paper)</th>
  </tr>
  <tr>
    <td>Original</td><td>54.7</td><td>53.0</td><td>50.2</td><td>52.6</td><td>54.8</td><td>52.6</td><td>52.8</td><td>52.4</td><td>51.0</td><td>52.7</td><td>55.6</td><td>51.5</td><td>52.4</td><td>52.8</td><td>52.1</td><td>52.3</td><td>53.0</td><td>52.8</td>
  </tr>
  <tr>
    <td>PCW</td><td>12.4</td><td>11.9</td><td>12.2</td><td>12.2</td><td>3.7</td><td>4.0</td><td>4.0</td><td>4.0</td><td>3.9</td><td>3.9</td><td>2.3</td><td>1.8</td><td>2.0</td><td>2.0</td><td>2.1</td><td>2.0</td><td>2.0</td><td>2.0</td>
  </tr>
  <tr>
    <td>Set-Based Prompting</td><td>42.5</td><td>42.5</td><td>42.5</td><td>42.5</td><td>26.3</td><td>26.3</td><td>26.3</td><td>26.3</td><td>26.3</td><td>26.3</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td><td>14.1</td>
  </tr>
  <tr>
    <td>PINE</td><td>58.8</td><td>58.8</td><td>59.0</td><td>58.8</td><td>56.2</td><td>55.7</td><td>55.5</td><td>55.7</td><td>55.5</td><td>55.7</td><td>54.2</td><td>54.8</td><td>54.3</td><td>53.7</td><td>54.8</td><td>54.2</td><td>54.0</td><td>54.3</td>
  </tr>
  <tr>
    <td>RoToR-lexical</td><td>61.4</td><td>61.6</td><td>61.6</td><td>61.5</td><td><b>61.4</b></td><td>59.8</td><td>59.6</td><td>59.6</td><td>59.8</td><td>60.0</td><td>59.2</td><td>59.5</td><td>59.4</td><td>59.1</td><td>59.0</td><td>59.3</td><td>59.1</td><td>59.2</td>
  </tr>
  <tr>
    <td>RoToR-reversed lexical</td><td><b>61.6</b></td><td><b>61.8</b></td><td><b>61.8</b></td><td><b>61.8</b></td><td>58.9</td><td>59.3</td><td>58.8</td><td>58.6</td><td>58.7</td><td>58.8</td><td>57.9</td><td>58.2</td><td>57.9</td><td>57.4</td><td>57.9</td><td>57.6</td><td>57.5</td><td>57.8</td>
  </tr>
  <tr>
    <td>RoToR-reranking</td><td>61.2</td><td>61.4</td><td>61.2</td><td>61.3</td><td>60.9</td><td>61.0</td><td><b>61.2</b></td><td><b>61.2</b></td><td><b>61.2</b></td><td><b>61.1</b></td><td><b>60.9</b></td><b>60.7</b><td>60.7</td><td>60.7</td><td>60.8</td><td>60.8</td><td>60.8</td><td>60.8</td>
  </tr>
  <tr>
    <td>RoToR-freq</td><td>61.0</td><td>61.1</td><td>61.1</td><td>61.1</td><td>60.4</td><td>60.3</td><td>58.6</td><td>60.2</td><td>60.0</td><td>59.9</td><td>59.3</td><td>60.4</td><td>59.7</td><td>59.5</td><td>59.5</td><td>59.6</td><td>59.2</td><td>59.6</td>
  </tr>
</table>

Table 5: The best_subspan_em (%) scores on the lost in the middle (LitM) benchmark. For RoToR, we test three different global ordering strategies (lexical, reversed lexical, and MonoT5-base reranking) across varying numbers of documents (ndoc \( \in \{10, 20, 30\} \)). Appendix Fig. 5 visualizes the fluctuations across different gold positions. RoToR shows the best performance across all setups, and is especially more stable when indexing bias in the input is removed.

Different global ordering strategies given example input: 7 rows

0: row content : direct war losses | 360,000 | null | null | null | 183,000 | 543,000!!!!!!!!!!!!
1: row content : murdered | 75,000 | 100,000 | 118,000 | 133,000 | 82,000 | null | 506,000!!!!
2: row content : deaths in prisons & camps | 69,000 | 210,000 | 220,000 | 266,000 | ...
3: row content : deaths outside of prisons & camps | null | 42,000 | 71,000 | 142,000 | ...
4: row content : murdered in eastern regions | null | null | null | null | 100,000 | 100,000!!!!!!!!!!!!
5: row content : deaths other countries | null | null | null | null | null | 2,000,000!!!!
6: row content : total | 504,000 | 352,000 | 407,000 | 541,000 | 681,000 | 2,770,000!

Tokenized IDs
(1) Global sort as lexical sort: 0>6>2>5>3>4>1
(2) Global sort as reranking sort:
0: 0.1 6>5>2>4>1>0>3
1: 0.15
2: 0.4
3: 0.05
4: 0.2 p(Yes) + p(No)
5: 0.6
6: 0.9

Frequency Count:
[765; 10, 198; 7, 654; 2, 262; 5, 551; 7, 220; 3, 11; 3, 1679; 3, 854; 3, 32485; 2, 931;...]
(3) Global sort as frequency-based sort: 2>5>3>1>4>6>1
Divide by Frequency
0 [1/7, 1/7, 1/7, 1/7, 1/1, 1/1, 1/10, 1/3, 1/1, 1/3, ...]
1 [1/7, 1/7, 1/7, 1/7, 1/2, 1/10, 1/3, 1/1, 1/3, 1/10, ...]
2 [1/7, 1/7, 1/7, 1/7, 1/3, 1/2, 1/2, 1/2, 1/10, 1/1, ...]
3 [1/7, 1/7, 1/7, 1/7, 1/3, 1/1, 1/1, 1/2, 1/2, 1/10, ...]
4 [1/7, 1/7, 1/7, 1/7, 1/2, 1/2, 1/1, 1/10, 1/3, 1/1, ...]
5 [1/7, 1/7, 1/7, 1/7, 1/3, 1/1, 1/1, 1/10, 1/3, 1/1, ...]
6 [1/7, 1/7, 1/7, 1/7, 1/1, 1/10, 1/3, 1/1, 1/3, 1/10, ...]

Query: what is the total number of murdered?, Document: row content : direct war losses | 360... 3,000!!!!!!!!!!!!, Relevant: ...
Query: what is the total number of murdered?, Document: row content : total | 504,000 | 35...000 | 2,770,000!, Relevant:

Reranker (MonoT5-base)

Figure 6: Illustration of ordering 7 rows by 2 different global sort options, (1) lexical sort based on token ids, or (2) reranking sort based on a reranker model (MonoT5-base in this case). (3) frequency based sorting applies lexical sorting, but the definitive token ids are mapped into the inverse frequency.

for all experiments. For all datasets we tested, we separate the input to 3 parts: prefix, parallel contexts, and suffix and feed them accordingly to the positional-invariant model. For the case of the original model, we simply join the prefix, context, and suffix text to make one sequential text. For testing with the PCW model, we concat each prefix to the parallel context due to the architectural limitations of the PCW model, which doesn’t have the prefix part (which is processed casually before parallel contexts). For example, on testing the PCW model on MMLU, we append the question and each answer options, which generally result in longer input sequences. For PCW, we use the pcw_generate function, and we additionally
<table>
  <tr>
    <th rowspan="2">Validation set (1531)</th>
    <th colspan="10">α = | no Selective Routing -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5</th>
  </tr>
  <tr>
    <th colspan="6">&lt;-- bias towards invariant model</th>
    <th colspan="5">bias towards orig model --&gt;</th>
  </tr>
  <tr>
    <td>Selective Routing (orig, pine)</td>
    <td>64.9</td><td>65.3</td><td>65.8</td><td>66.4</td><td>67.3</td><td>67.4</td><td>67.6</td><td>67.5</td><td>67.9</td><td>67.8</td><td>68.0</td><td>67.9</td>
  </tr>
  <tr>
    <td>Selective Routing (orig, ours-lexical)</td>
    <td>63.9</td><td>64.1</td><td>64.5</td><td>65.4</td><td>65.8</td><td>67.0</td><td>67.4</td><td>68.1</td><td>68.2</td><td>68.1</td><td>67.9</td><td>67.9</td>
  </tr>
  <tr>
    <td>Selective Routing (orig, ours-monot5)</td>
    <td>66.0</td><td>66.1</td><td>66.3</td><td>66.7</td><td>67.0</td><td>67.6</td><td>68.0</td><td>68.1</td><td>68.1</td><td>67.7</td><td>67.9</td><td>67.9</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Test set (14015)</th>
    <th colspan="10">α = | no Selective Routing -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5</th>
  </tr>
  <tr>
    <th colspan="6">&lt;-- bias towards invariant model</th>
    <th colspan="5">bias towards orig model --&gt;</th>
  </tr>
  <tr>
    <td>Selective Routing (orig, pine)</td>
    <td>64.8</td><td>64.9</td><td>65.4</td><td>66.0</td><td>66.8</td><td>67.5</td><td>68.4</td><td>68.5</td><td>68.5</td><td>68.4</td><td>68.3</td><td>68.3</td>
  </tr>
  <tr>
    <td>Selective Routing (orig, ours-lexical)</td>
    <td>63.2</td><td>63.5</td><td>64.2</td><td>65.2</td><td>66.2</td><td>67.3</td><td>68.0</td><td>68.4</td><td>68.5</td><td>68.5</td><td>68.4</td><td>68.3</td>
  </tr>
  <tr>
    <td>Selective Routing (orig, ours-monot5)</td>
    <td>64.2</td><td>64.4</td><td>64.8</td><td>65.5</td><td>66.4</td><td>67.3</td><td>68.1</td><td>68.5</td><td>68.4</td><td>68.5</td><td>68.3</td><td>68.3</td>
  </tr>
</table>

Table 6: Reporting full ablation results on application of Selective Routing. \( \alpha = 0.2 \) was the best for the validation set, which was then applied to obtain the reported results for all models.

utilized the RestrictiveTokensLogitsProcessor provided at the official PCW repository for MMLU classification, to have a similar setup with the log_likelihood option used for other models.

C.2 Knowledge Graph Question Answering

For evaluation with Mintaka, we follow the same setup as Baek et al. (2023). Given the gold answer and model generated answer, the EM score counts if both are exactly the same; Accuracy measures if the generated answer includes the gold answer, and F1 score measures the precision and recall among overlapping words. Since we are testing on a non-trained zero-shot version of the model, we enforce the model to output in json format to make it easier to parse. For the row shuffling setup, we fix the seed to 0, 1, 2 on shuffling rows and report the average scores.

C.3 Lost in the Middle

Specifically, we use the dataset provided in the official repository\footnote{github.com/nelson-liu/lost-in-the-middle}, use the same prompt as the llama 2 chat model with only the instruction tokens adjusted to llama 3 (removed [Inst] and changed to 
 and etc..), and evaluate using the best_supspan_em metric.

C.4 MMLU

We follow the publicly acknowledged lm-evaluation harness (Gao et al., 2024) prompt design by cluther.ai. We measure accuracy between the gold answer and the token with the highest likelihood (probability) among possible answer tokens [ 'A', 'B', 'C', 'D'].

D Further impact scenarios on general conversation.

We shortly discuss about how this method may be applied to general conversational scenarios of LLMs. For processing contexts such as chronological history of conversations, the ordering is important, and the original LLM remains the better choice for this case. However, in subsets of conversational tasks requiring order invariance (e.g., Sets, Tables, or RAG contexts), our method enhances unbiased understanding, as demonstrated mainly in Lost-in-the-Middle benchmark. Here, RoToR achieves a significant 7-9% average accuracy gain over the original LLM, very consistently across all setups (doc indexing and ndoc) for all choices of the ordering algorithm, with lower standard deviation than the original model.

E Selection of \( \alpha \) for Selective Routing on MMLU

We report \( \alpha \) is a hyperparameter that can be tuned per-dataset. We searched its value in the range of -0.5 to 0.5 with a step size of 0.1 using the validation split of MMLU\footnote{https://huggingface.co/datasets/cais/mmlu/viewer/abstract_algebra/validation} on RoToR with lexical sorting, and applied the found value (0.2) on the test set to obtain the reported results for all models. We report the full variation of Selective Routing results on the investigated \( \alpha \) value at Tab. 6.
F  Replication details on the runtime experiment

Apart from the theoretical runtime efficiency, we measured the actual end-to-end runtime in seconds, to better analyze the practical runtime efficiency between PINE and ROToR. The runtime of each experiment was measured on an ASUS ESC8000-E11 server featuring dual 4th Gen Intel Xeon Scalable processors, 64 CPU threads, 1.1 TB of RAM across 32 DIMM slots, and 8 NVIDIA A6000 GPUs with 48 GB of memory each. We Except for the experiments on Llama-3.1-70B-Instruct, we only use a single A6000 GPU for all of the experiments.

G  Input data examples

To illustrate the input and output formats used in our experiments, we provide example inputs for the Lost-in-the-Middle (LitM), Knowledge Graph Question Answering (KGQA), and MMLU datasets. For experiments using the Qwen-Chat model, special tokens were adjusted accordingly. While the example prompts are based on the Llama-3.1-8B-Instruct model, the specific differences in token usage for the Qwen-Chat variants can be observed by comparing the prompts in Fig. 8 and Fig. 9. This adjustment is consistently applied across all datasets. Note that no special tokens are added for the MMLU benchmark, which aligns with the lm-evaluation harness setup.

<table>
  <tr>
    <th colspan="2">lost in the middle</th>
  </tr>
  <tr>
    <th>Prefix:</th>
    <td>&lt;begin_of_text&gt;&lt;lstart_header_idl&gt;system&lt;lend_header_idl&gt;</td>
  </tr>
  <tr>
    <td colspan="2">
      You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don’t share false information.&lt;leot_idl&gt;&lt;lstart_header_idl&gt;user&lt;lend_header_idl&gt;
    </td>
  </tr>
  <tr>
    <td colspan="2">
      Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant).
    </td>
  </tr>
  <tr>
    <th>Parallel texts:</th>
    <td>
      Document [1](Title: List of Nobel laureates in Physics) The first ...
      ...
      Document [10](Title: Nobel Prize in Chemistry) on December 10, the ...
    </td>
  </tr>
  <tr>
    <th>Suffix:</th>
    <td>
      Question: who got the first nobel prize in physics&lt;leot_idl&gt;&lt;lstart_header_idl&gt;assistant &lt;lend_header_idl&gt;
    </td>
  </tr>
</table>

Figure 7: Example input for the lost in the middle dataset.
lost in the middle no indexing

Prefix:
<begin_of_text><lstart_header_idl>system<lend_header_idl>

You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don’t share false information.<leot_idl><lstart_header_idl>user<lend_header_idl>

Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant).

Parallel texts:
[Document Title: List of Nobel laureates in Physics.] The first ...
...
[Document Title: Nobel Prize in Chemistry.] on December 10, the ...

Suffix:
Question: who got the first nobel prize in physics<leot_idl><lstart_header_idl>assistant<lend_header_idl>

Figure 8: Example input for the lost in the middle dataset, without indexing by numbers. Prompt for the Llama-3.1-8B-Instruct model.

lost in the middle no indexing (Qwen variant)

Prefix:
<lim_start>system
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don’t share false information.<lim_end><lim_start>user

Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant).

Parallel texts:
[Document Title: Thorax] when deep breaths are attempted. Different people ...
...
[Document Title: Chest pain] present with chest pain, and carry a significantly higher ...

Suffix:
Question: for complaints of sudden chest pain patients should take a<lim_end>
<lim_start>assistant

Figure 9: Example input for the lost in the middle dataset, without indexing by numbers, prompt for the Qwen1.5-Chat model.
Mintaka

Prefix:
<begin_of_text><lstart_header_idl>system<lend_header_idl>

Below are the facts in the form of the triple meaningful to answer the question. Answer the given question in a JSON format, such as "Answer": "xxx". Only output the JSON, do NOT say any word or explain.

<leot_idl><lstart_header_idl>user<lend_header_idl>

Parallel texts:
(Super Bowl XLII, winner, New York Giants)
(Super Bowl XLII, participating team, New York Giants)
(Super Bowl XLII, point in time, time: +2008-02-03)
(Super Bowl XLII, followed by, Super Bowl XLIII)
(Super Bowl XLII, location, State Farm Stadium)
...
(Super Bowl XLII, sport, American football)
(Super Bowl XLII, instance of, Super Bowl)

Suffix:
Question: which team did the super bowl xlii mvp play for?, Answer: <leot_idl><lstart_header_idl>assistant <lend_header_idl>

Gold Answer(s):
('NYG', 'Giants', 'NY Giants', 'New York Giants')

Example generated output:
{"Answer": "New York Giants"} (Parsed to: New York Giants)

Figure 10: Example input for the Mintaka dataset.

MMLU

Prefix:
The following are multiple choice questions (with answers) about moral disputes.

Norcross agrees that if a being is incapable of moral reasoning, at even the most basic level, then it cannot be

Parallel texts:
A. a being of value.
B. an object of moral sympathy.
C. a moral agent.
D. a moral patient.

Suffix:
Answer:

Figure 11: Example input for the MMLU benchmark.
H  Why is removing index bias an important detail for invariant models to be effective?

The alphabetic index (A/B/C/D) introduced in Fig. 1 associated with each segment, reportedly introduces token bias (Wei et al., 2024) of preferring the choice marked as ‘A.’ The same thing can be applied to listwise inputs with simple numeric indexing (1/2/3/4), which was the case for the lost-in-the middle benchmark. While a standard model with no modifications on positional encoding correctly places contexts indexed A before contexts indexed with D by positional encoding, an invariant model sees contexts in an order-agnostic way, meaning that the alphabetical indexing may not always be interpreted sequentially and thus can confuse the model from accurately interpreting the contexts. For example, even for cases where the index ordering of the input was in alphabetical order (A->B->C->D), the ordering-invariant model may interpret contexts with (C->A->B->D) at one point (e.g., when the query is D on self attention), which can cause unnatural, out-of-distribution representation, leading to decreased performance.

I  Statistical significance before and after shuffling segments

We conducted paired two-tailed t-tests (Table 8) for both the baseline (“original”) model and our proposed method (ROToR), using the results in Table 7. Our goal was to determine whether the performance differences between the initial ordering and shuffled ordering are statistically significant. We excluded the Lost-in-the-Middle (LitM) dataset because it does not provide an initial ordering. Specifically, the tests evaluate whether the mean performance difference (Before Shuffle - After Shuffle) significantly deviates from zero.

For KGQA, we selected the F1 score as the representative metric among the three available, gathering data points from various task configurations and different models. For MMLU, the results are based on our RoToR variant with selective routing. As shown in Table 8, the original model shows a statistically significant drop in performance when the segments are shuffled, while RoToR does not, indicating increased robustness to segment-order perturbations. 11

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Original Model</th>
    <th colspan="3">RoToR-lexical</th>
  </tr>
  <tr>
    <th>Before Shuff.</th>
    <th>After Shuff.</th>
    <th>Diff.</th>
    <th>Before Shuff.</th>
    <th>After Shuff.</th>
    <th>Diff.</th>
  </tr>
  <tr>
    <td>Mintaka, Llama3.1-8B-Instruct, ndoc=30</td>
    <td>51.9</td>
    <td>51.0</td>
    <td>0.9</td>
    <td>54.1</td>
    <td>53.8</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Mintaka, Llama3.1-8B-Instruct, ndoc=50</td>
    <td>51.7</td>
    <td>51.0</td>
    <td>0.7</td>
    <td>53.6</td>
    <td>53.5</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Mintaka, Qwen1.5-4B-Chat, ndoc=30</td>
    <td>34.9</td>
    <td>34.7</td>
    <td>0.2</td>
    <td>35.7</td>
    <td>35.5</td>
    <td>0.2</td>
  </tr>
  <tr>
    <td>Mintaka, Qwen1.5-4B-Chat, ndoc=50</td>
    <td>35.8</td>
    <td>35.0</td>
    <td>0.8</td>
    <td>36.2</td>
    <td>36.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Mintaka, Qwen1.5-7B-Chat, ndoc=30</td>
    <td>35.4</td>
    <td>35.0</td>
    <td>0.4</td>
    <td>37.7</td>
    <td>37.7</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Mintaka, Qwen1.5-7B-Chat, ndoc=50</td>
    <td>35.7</td>
    <td>35.5</td>
    <td>0.2</td>
    <td>38.0</td>
    <td>38.0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MMLU, Llama3.1-8B-Instruct</td>
    <td>68.3</td>
    <td>65.5</td>
    <td>2.8</td>
    <td>68.5</td>
    <td>65.7</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>MMLU, Qwen1.5-4B-Chat</td>
    <td>53.6</td>
    <td>52.6</td>
    <td>1</td>
    <td>53.7</td>
    <td>52.6</td>
    <td>1.1</td>
  </tr>
  <tr>
    <td>MMLU, Qwen1.5-7B-Chat</td>
    <td>60.1</td>
    <td>58.6</td>
    <td>1.5</td>
    <td>60.1</td>
    <td>58.8</td>
    <td>1.3</td>
  </tr>
</table>

Table 7: Performance of the Original model and RoToR before and after shuffling.

Derivation for the original model. Let the nine paired differences (Before – After) be \( \{d_1, d_2, d_3, ... d_8, d_9\} \). Mean Difference: \( \bar{d} = \frac{1}{9} \sum_{i=1}^9 d_i \). In this case, \( \bar{d} \approx 0.9444\% \). Sample Standard Deviation: \( s_d = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (d_i - \bar{d})^2} \approx 0.7632 \). Standard Error (SE): SE = \( \frac{s_d}{\sqrt{n}} \approx 0.2544 \). t-Statistic: \( t = \frac{\bar{d}}{\text{SE}} \approx 3.7124 \), (\( df = 8 \)). Since the critical value at \( df = 8 \) and \( \alpha = 0.05 \) is 2.306, we have \( 3.71 > 2.306 \). Therefore, the difference is statistically significant.

11 All statistical calculations were validated using an online t-test calculator: https://www.mathportal.org/calculators/statistics-calculator/t-test-calculator.php
<table>
  <tr>
    <th></th>
    <th>Original</th>
    <th>RoToR</th>
  </tr>
  <tr>
    <td>Degrees of Freedom</td>
    <td>8</td>
    <td></td>
  </tr>
  <tr>
    <td>Mean Difference</td>
    <td>0.94</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td><i>t</i>-Statistic</td>
    <td>3.71</td>
    <td>2.23</td>
  </tr>
  <tr>
    <td>Critical Value</td>
    <td colspan="2">2.306</td>
  </tr>
  <tr>
    <td>Statistically Significant</td>
    <td><b>Yes</b></td>
    <td><b>No</b></td>
  </tr>
</table>

Table 8: Paired two-tailed t-test results comparing the original model and ours.

Derivation for the RoToR model. Under the same procedure, \( \bar{d} \approx 0.6556\%, \ s_d \approx 0.8833,\ SE \approx 0.2944 \). <i>t</i>-<b>Statistic:</b> \( t \approx 2.2265 \). Since \( 2.2265 < 2.306 \), there is no significant difference in performance before and after shuffling for RoToR.

Conclusion. While the original model shows a statistically significant performance drop with shuffled inputs, RoToR remains unaffected, demonstrating greater robustness to segment-order perturbations.

J Application to Long-Context Inputs

Benchmark and protocol. To test whether RoToR consistantly performs well to inputs with longer contexts, we extend our evaluation to the LongBench (Bai et al., 2024)–2WikiMultihopQA task, whose multi-document questions yield input lengths from 5 k to 15 k tokens. Because our current RoToR and PINE implementations do not yet support advanced parallelization (e.g., FlashAttention or SDPA), GPU memory becomes prohibitive beyond ~10 k tokens, especially for PINE, whose memory footprint is larger than ours. We therefore truncate the context at 10 k tokens, which already exceeds the lengths used in our main experiments.

Following the official LongBench guidelines\footnote{Maximum generation length 32 and qa\_f1\_score (LongBench-E) as the evaluation metric.}, we segment each long context into 512-token “listwise” inputs, enabling a direct comparison with our listwise reranking pipeline. To ensure robustness, we test three input-order perturbations:

1. <b>Initial:</b> original chunk order [1, 2, 3, 4, 5];
2. <b>Reversed:</b> [5, 4, 3, 2, 1];
3. <b>Center-flipped:</b> first and last halves swapped, [3,2,1,5,4].

Results. Table 9 reports F1 scores for the Original model variant, RoToR, and PINE with <b>Llama 3.1-8B-Instruct</b> and <b>Qwen 1.5-7B-Chat</b>. Across every ordering and token-length band, RoToR retains a clear advantage, while PINE fails to run when the longest (8 k+) chunks are kept. These findings confirm that RoToR scales to substantially longer inputs across different input-order perturbations.

Take-aways. Even without specialised long-context kernels, RoToR consistently outperforms both the original reranker and PINE, and remains robust to severe input-order perturbations. This suggests our approach can generalize to substantially longer inputs once memory and kernel constraints are alleviated.

K Robustness to task templates

We evaluated whether the proposed selective-routing methods remain effective when the surrounding task instructions are re-phrased. Concretely, we performed a <i>template-swap</i> experiment on the KGQA benchmark, specifically on the initial ordering setup. The original prompt (Figure 10) began with

“Below are the facts in the form of triples meaningful to answer the question.”
<table>
  <tr>
    <th rowspan="2">Order</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Llama 3.1-8B-Instruct</th>
    <th colspan="4">Qwen 1.5-7B-Chat</th>
  </tr>
  <tr>
    <th>0–4k</th>
    <th>4–8k</th>
    <th>8k+</th>
    <th>Total</th>
    <th>0–4k</th>
    <th>4–8k</th>
    <th>8k+</th>
    <th>Total</th>
  </tr>
  <tr>
    <td colspan="10">Count</td>
    <td>25</td>
    <td>131</td>
    <td>144</td>
    <td>300</td>
    <td>23</td>
    <td>121</td>
    <td>156</td>
    <td>300</td>
  </tr>
  <tr>
    <td colspan="10">Initial (e.g., Orig., PINE, RoToR)</td>
    <td>48.3</td>
    <td>56.8</td>
    <td>34.0</td>
    <td>45.1</td>
    <td>65.6</td>
    <td>47.9</td>
    <td>26.7</td>
    <td>38.2</td>
  </tr>
  <tr>
    <td colspan="10">Reverse (e.g., Orig., PINE, RoToR)</td>
    <td>51.0</td>
    <td>47.6</td>
    <td>–</td>
    <td>–</td>
    <td>70.2</td>
    <td>45.1</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td colspan="10">Reverse (e.g., Orig., PINE, RoToR)</td>
    <td>59.0</td>
    <td>52.7</td>
    <td>41.8</td>
    <td>48.0</td>
    <td>75.7</td>
    <td>47.8</td>
    <td>31.0</td>
    <td>41.2</td>
  </tr>
  <tr>
    <td colspan="10">Center flip (e.g., Orig., PINE, RoToR)</td>
    <td>57.0</td>
    <td>51.5</td>
    <td>39.0</td>
    <td>46.0</td>
    <td>53.4</td>
    <td>43.3</td>
    <td>34.2</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td colspan="10">Center flip (e.g., Orig., PINE, RoToR)</td>
    <td>43.0</td>
    <td>49.8</td>
    <td>–</td>
    <td>–</td>
    <td>64.1</td>
    <td>48.9</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td colspan="10">Center flip (e.g., Orig., PINE, RoToR)</td>
    <td>59.0</td>
    <td>52.0</td>
    <td>41.0</td>
    <td>47.3</td>
    <td>72.8</td>
    <td>47.6</td>
    <td>30.8</td>
    <td>40.8</td>
  </tr>
  <tr>
    <td colspan="10">Center flip (e.g., Orig., PINE, RoToR)</td>
    <td>47.0</td>
    <td>47.7</td>
    <td>35.6</td>
    <td>41.8</td>
    <td>61.0</td>
    <td>40.6</td>
    <td>32.7</td>
    <td>38.1</td>
  </tr>
  <tr>
    <td colspan="10">Center flip (e.g., Orig., PINE, RoToR)</td>
    <td>46.3</td>
    <td>49.2</td>
    <td>–</td>
    <td>–</td>
    <td>70.2</td>
    <td>43.5</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td colspan="10">Center flip (e.g., Orig., PINE, RoToR)</td>
    <td>59.0</td>
    <td>52.5</td>
    <td>41.5</td>
    <td>47.8</td>
    <td>77.1</td>
    <td>47.3</td>
    <td>30.9</td>
    <td>41.0</td>
  </tr>
</table>

Table 9: F1 scores (%) on LONGBENCH–2WikiMultihopQA with ~10k-token contexts. “Count” is the number of examples per length bucket; “–” denotes out-of-memory.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Llama-3.1-8B-Instruct</th>
    <th colspan="3">Qwen-1.5-4B-Chat</th>
    <th colspan="3">Qwen-1.5-7B-Chat</th>
  </tr>
  <tr>
    <th>N = 30</th>
    <th>N = 50</th>
    <th>N = 30</th>
    <th>N = 50</th>
    <th>N = 30</th>
    <th>N = 50</th>
  </tr>
  <tr>
    <td>Original template, Initial ordering</td>
    <td>50.2</td>
    <td>44.0</td>
    <td>51.9</td>
    <td>50.0</td>
    <td>44.0</td>
    <td>51.7</td>
    <td>30.7</td>
    <td>27.9</td>
    <td>34.9</td>
    <td>31.6</td>
    <td>28.6</td>
    <td>35.8</td>
    <td>31.5</td>
    <td>27.8</td>
    <td>35.4</td>
    <td>31.7</td>
    <td>28.0</td>
    <td>35.7</td>
  </tr>
  <tr>
    <td>PINE</td>
    <td>51.5</td>
    <td>45.0</td>
    <td>52.6</td>
    <td>51.6</td>
    <td>45.1</td>
    <td>52.6</td>
    <td>31.6</td>
    <td>28.7</td>
    <td>35.6</td>
    <td>31.6</td>
    <td>28.8</td>
    <td>35.3</td>
    <td>32.3</td>
    <td>28.8</td>
    <td>36.4</td>
    <td>32.0</td>
    <td>28.5</td>
    <td>35.9</td>
  </tr>
  <tr>
    <td>RoToR</td>
    <td><b>53.1</b></td>
    <td><b>46.5</b></td>
    <td><b>54.1</b></td>
    <td>52.9</td>
    <td>46.0</td>
    <td>53.6</td>
    <td>32.0</td>
    <td>29.0</td>
    <td>35.7</td>
    <td><b>32.7</b></td>
    <td><b>29.6</b></td>
    <td><b>36.2</b></td>
    <td><b>34.3</b></td>
    <td><b>29.8</b></td>
    <td><b>37.7</b></td>
    <td><b>34.3</b></td>
    <td><b>30.1</b></td>
    <td><b>38.0</b></td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td>
    <td>51.6</td>
    <td>45.0</td>
    <td>52.5</td>
    <td>52.4</td>
    <td>45.4</td>
    <td>52.8</td>
    <td>32.3</td>
    <td>29.1</td>
    <td>36.2</td>
    <td>32.3</td>
    <td>29.3</td>
    <td>35.9</td>
    <td>32.9</td>
    <td>28.4</td>
    <td>36.3</td>
    <td>32.9</td>
    <td>28.9</td>
    <td>36.6</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td>
    <td>52.6</td>
    <td>46.1</td>
    <td>53.7</td>
    <td><b>53.1</b></td>
    <td><b>46.4</b></td>
    <td><b>53.7</b></td>
    <td><b>32.3</b></td>
    <td><b>29.2</b></td>
    <td><b>36.0</b></td>
    <td>32.3</td>
    <td>29.2</td>
    <td>35.9</td>
    <td>33.7</td>
    <td>29.5</td>
    <td>37.2</td>
    <td>33.5</td>
    <td>29.5</td>
    <td>37.2</td>
  </tr>
  <tr>
    <td>Template-swap</td>
    <td>50.0</td>
    <td>44.1</td>
    <td>51.8</td>
    <td>50.2</td>
    <td>44.3</td>
    <td>51.9</td>
    <td>31.1</td>
    <td>27.8</td>
    <td>34.9</td>
    <td>31.7</td>
    <td>28.3</td>
    <td>35.3</td>
    <td>31.4</td>
    <td>27.6</td>
    <td>35.2</td>
    <td>32.0</td>
    <td>28.0</td>
    <td>35.7</td>
  </tr>
  <tr>
    <td>PINE</td>
    <td>52.0</td>
    <td>45.7</td>
    <td>52.9</td>
    <td>52.0</td>
    <td>45.3</td>
    <td>52.8</td>
    <td>31.9</td>
    <td>28.8</td>
    <td>35.8</td>
    <td>31.7</td>
    <td>28.6</td>
    <td>35.4</td>
    <td>31.9</td>
    <td>28.5</td>
    <td>36.0</td>
    <td>31.5</td>
    <td>28.2</td>
    <td>35.7</td>
  </tr>
  <tr>
    <td>RoToR</td>
    <td><b>52.7</b></td>
    <td><b>46.4</b></td>
    <td><b>54.0</b></td>
    <td><b>52.9</b></td>
    <td><b>46.4</b></td>
    <td><b>53.7</b></td>
    <td>31.8</td>
    <td>28.2</td>
    <td>35.0</td>
    <td>32.4</td>
    <td>29.0</td>
    <td>35.6</td>
    <td><b>34.1</b></td>
    <td><b>29.8</b></td>
    <td><b>37.6</b></td>
    <td><b>34.0</b></td>
    <td><b>29.9</b></td>
    <td><b>37.7</b></td>
  </tr>
  <tr>
    <td>RoToR-MonoT5</td>
    <td>51.5</td>
    <td>45.2</td>
    <td>52.6</td>
    <td>52.5</td>
    <td>45.7</td>
    <td>53.1</td>
    <td><b>32.4</b></td>
    <td><b>29.0</b></td>
    <td><b>36.3</b></td>
    <td><b>32.6</b></td>
    <td><b>29.3</b></td>
    <td><b>35.9</b></td>
    <td>32.9</td>
    <td>28.5</td>
    <td>36.4</td>
    <td>32.6</td>
    <td>28.5</td>
    <td>36.3</td>
  </tr>
  <tr>
    <td>RoToR-Freq.</td>
    <td>52.3</td>
    <td>46.2</td>
    <td>53.7</td>
    <td><b>52.9</b></td>
    <td><b>46.5</b></td>
    <td><b>53.8</b></td>
    <td>31.9</td>
    <td>28.4</td>
    <td>35.4</td>
    <td>32.4</td>
    <td>28.8</td>
    <td>35.6</td>
    <td>34.0</td>
    <td><b>29.9</b></td>
    <td>37.4</td>
    <td>33.8</td>
    <td>29.6</td>
    <td>37.4</td>
  </tr>
</table>

Table 10: Results on the Mintaka (KGQA) dataset on different models, before (top block, also reported at main paper) and after (bottom block) the template-swap. N refers to number of top-k segments per query. RoToR variants consistently outperform the Original and PINE baselines, and their performance is stable under the swapped template, indicating robustness to instruction wording.

and required the model to output only a JSON object. In the swapped template we replaced the first sentence with

“Below are knowledge statements expressed as triples meaningful to answer the question.”

leaving all other instructions unchanged. Table 10 reports the results. Across all three backbone models and both retrieval depths (\(N=30, 50\)), RoToR and its variants retain similar absolute scores and continue to outperform both the Original and PINE baselines, indicating strong robustness to superficial wording changes in the task template.

L Additional Statistics on Selective Routing Assignment

Table 11 complements the main results in Table 3 by reporting the selection ratio, accounting for the percentage of evaluation queries for which the RoToR branch is chosen over the vanilla branch, under Selective Routing (SR).13 We break the analysis down by (i) the global sorting strategy (Lexical, MonoT5, or Freq.), and (ii) three model backbones. The table distinguishes three order–based conditions:

For table 11, Init. refers to the original ordering (e.g., in abcd order), Rev. refers to the reversed ordering (e.g., in dcba but assigned as abcd), and Avg. is the average selection ratio for all possible (4!-1) re-orderings, with standard deviation. Empirically, the RoToR model tends to be selected more frequently

13 All figures are computed over the full evaluation set of 14,015 queries.
<table>
  <tr>
    <th rowspan="2">Sorting</th>
    <th colspan="3">Llama-3.1-8B-Instr.</th>
    <th colspan="3">Qwen1.5-4B-Chat</th>
    <th colspan="3">Qwen1.5-7B-Chat</th>
  </tr>
  <tr>
    <th>Init.</th>
    <th>Rev.</th>
    <th>Avg.</th>
    <th>Init.</th>
    <th>Rev.</th>
    <th>Avg.</th>
    <th>Init.</th>
    <th>Rev.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Lexical</td>
    <td>7.0</td>
    <td>8.5</td>
    <td>7.3±0.8</td>
    <td>5.9</td>
    <td>6.2</td>
    <td>6.2±0.4</td>
    <td>10.3</td>
    <td>10.6</td>
    <td>9.9±0.6</td>
  </tr>
  <tr>
    <td>MonoT5</td>
    <td>6.9</td>
    <td>7.6</td>
    <td>6.7±1.5</td>
    <td>8.0</td>
    <td>12.5</td>
    <td>9.8±2.1</td>
    <td>10.7</td>
    <td>10.9</td>
    <td>10.7±0.7</td>
  </tr>
  <tr>
    <td>Freq.</td>
    <td>6.4</td>
    <td>6.7</td>
    <td>6.9±0.5</td>
    <td>8.5</td>
    <td>10.9</td>
    <td>9.4±1.6</td>
    <td>10.7</td>
    <td>11.1</td>
    <td>11.1±0.8</td>
  </tr>
</table>

Table 11: Selection ratio (%) of the RoToR variant under SR. Higher values indicate more frequent routing to RoToR.

under reversed orderings, whereas under the original ordering, the vanilla model is chosen slightly more often. The exact ratio varies by model and sorting strategy.

Reviewer paper summaries: This paper targets the problem of positional bias in LLMs. The proposed RoToR can well handle this problem by identifying core issues in distribution mismatch and adaptive handling of mixed inputs.

This paper proposes RoToR, a stable zero-shot order-invariant solution that reduces the complexity of positional ID modification and Selective Routing that adaptively routes between invariant and sensitive LMs to handle real-world scenarios. Putting these two methods together, authors demonstrated improved performance and reliability on a list of well known MMLU benchmarks.
The RoToR is inspired by the word rotary to express circular assignment and also a palindrome to reflect order invariance.
Summary of strengths: 1. The studied problem is very important.
2. The method achieves superior performance.
3. The writing is easy to follow.

1. RoToR and selective routing are interesting solutions to deal with the lost-in-the-middle problem for KGQA, MMLU benchmarks.
2. detailed examples and solution are provided together with good results reported.
Summary of weaknesses: 1. The experiment is mainly conducted on relatively small LLMs, will the superiority be maintained with larger LLMs?
2. More experiments on the efficiency should be conducted.
3. Can the method be extended to other scenarios with no obvious Input Format Requirements?

1. Figure 2 and 3 are actually a bit difficult to follow and they reflect that your method is a bit fragile of too well-designed to be scable to large scale LLMs with super long context inputs.
2. better to see experiments of longer context inputs such as 128k to 1M to see how far your proposed methods can achieve. (such as scrolls, zero-scrolls, longbench and so on).
Comments, suggestions and typos: See weakness above

1. Figure 2 and 3 are actually a bit difficult to follow and they reflect that your method is a bit fragile of too well-designed to be scable to large scale LLMs with super long context inputs. You may consider to reduce the number of colors used - only the most important one shall be assigned with some color, otherwise it is easy to be lost learning your well-designed figures.

2. better to see experiments of longer context inputs such as 128k to 1M to see how far your proposed methods can achieve. (such as scrolls, zero-scrolls, longbench and so on). you can refer to these fact-based QA or summarization tasks to check if the global ordering really work or not when the context is extremely long.
Overall assessment scores: 3

3