Set-LLM: A Permutation-Invariant LLM
Authors: Beni Egressy
Jan Stühmer
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Large Language Models
LLM evaluator
LLM as a Judge
Permutation Invariance
Attention
Primary area: deep_learning
TL;DR: We introduce Set-LLM, a permutation-invariant LLM architecture that eliminates order bias and sensitivity.

Abstract: While large language models (LLMs) demonstrate impressive capabilities across numerous applications, their robustness remains a critical concern. This paper is motivated by a specific vulnerability: the order sensitivity of LLMs. This vulnerability manifests itself as the order bias observed when LLMs decide between possible options (for example, a preference for the first option) and the tendency of LLMs to provide different answers when options are reordered. The use cases for this scenario extend beyond the classical case of multiple-choice question answering to the use of LLMs for multidocument tasks and as automated evaluators in AI pipelines. We introduce Set-LLM, a novel architectural adaptation for pretrained LLMs that enables the processing of mixed set-text inputs with permutation invariance guarantees. The adaptations involve a new attention mask and new positional encodings specifically designed for sets. We provide a theoretical proof of invariance and demonstrate through experiments that Set-LLM can be trained effectively, achieving comparable or improved performance and maintaining the runtime of the original model, while altogether eliminating order sensitivity.

Set-LLM: A Permutation-Invarian LLM
Beni Egressy'
Jan Stühmer!
Heidelberg Institute for Theoretical Studies
IAR, Karlsruhe Institute of Technology
Abstract
While large language models (LLMs) demonstrate impressive capabilities across numerous applications, their robustness remains a critical concern. This paper is motivated by a specific vulnerability: the order sensitivity of LLMs. This vul- nerability manifests itself as the order bias observed when LLMs decide between possible options (for example, a preference for the first option) and the tendency of LL to provide different answers when options are reordered. The use cases for this scenario extend beyond the classical case of multiple-choice question an- swering to the use of LLMs for multidocument tasks and automated evaluators in AI pipelines. We introduce Set-LLM, a novel architectural adaptation for pre- trained LLMs that enables the processing of mixed set-text inputs with permutation invariance guarantees. The adaptations involve a new attention mask and new positional encodings specifically designed for sets. We provide theoretical proof invariance and demonstrate through experiments that Set-LLM can be trained effectively, achieving comparable or improved performance and maintaining the runtime of the original model, while altogether eliminating order sensitivity.
Introduction
The remarkable achievements of Large Language Models (LLMs) in recent years []] [6] have propelled their adoption across wide range of applications, including safety-critical and sensitive domains such as medicine and finance [25] [4]. As such, the eye-catching drops in performance from adversarial attacks can be all the more alarming B3 38 One such attack, shown in Figure [ is as trivial as permuting the choices in multiple-choice questions, which Zong et al. [5] demonstrate can degrade an LLM's 's performance from "good" worse than random.
This sensitivity to input order becomes even more critical given the increasing reliance on L.MMs to compare and evaluate the output of other L.MMs X 18
45 (3) Indeed, I.LM-as-a-judge is widely used as an evaluation metrie 2 53 and is also used to annotate LLM-generated output to create new data sets and decide between possible reasoning paths when solving complex problems 10 12 26 This inherent order sensitivity directly undermines the reliability of these pipelines.
We propose S a perruutaiooninnariia LLM architecture that eliminates this problem entirely. Set-LLM guarantees consistent responses by building permutation invariance directly into the model architecture. Moreover, it achieves these guarantees while retaining improving performance set-input tasks.
Set-LLM is based on the well-known observation that the attention mechanism underpinning all of the recent L.LM architectures is permutation invariant by construction. In fact, to force them to take into account the order of the input tokens, almost all models use some form of positional encoding However, Kazemnejad et al. [17] prove that even without positional encodings, the causal attention mask used in decoder-only LLMs is sufficient to completely reconstruct the input order. We therefore remove positional encoding and causal masks as the first steps towards Set-LLM.
37 39
All code is available under open licenses wtttttss://gtthhe httppp////ggaab.oooo/otttt/eeeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Where the Elffel Tower?
Where is the Elffll Tower?
Where the Elffei Tower?
Paris
Heidelberg
Set-LUM Adaptations:
Heide
Heidelberg
Provebly Permutation
Dudapest
Paris
Zurtch
Budapast
BU BuDipeen
Set Attention Mask (SetMazkk) Encoding (Se(PE)
Answer: Paris
Answer: Heidelbberg
Answer: Paris
An example of the vulnerability of LLMs to choice permutations. The LLM's response changes simply due to a reordering of the answer options. (Example for illustrative purposes only.) Set-LLM eliminates this vulnerability by building invariance directly into the model architecture.
Figure 1:
Our complete approach comprises four steps: (I) Removing sequential position encoding. (2) Remov- ing the causal mask, (3) Adding permutation-inee invvrriant set position encoding (SetPE), and (4) Adding a -ivaaraaann set attention mask (SetMask). These steps are illustrated in Figure 2 Together, the first two steps result in of-wodds (BoW) model, i.c., a model with no information about the token order, and the third and fourth steps add the order information we want back in.
We prove that Set-LLM is cemmutation-ivaaiiane meaning it is guaranteed to give the exact same output whatever input order is chosen. Moreover, demonstrate that the Set-LLM adaptations be combined with different decoder-only LLMs and do not depend on specific architectures or model versions, which can quickly become outdated. We experiments with five different base models on four multiple-choice datasets and two multi-document use cases to show the approach's 's versatility.
We summarize the main contributions follows:
We propose Set-LLM, the first permutation invariant decoder-only LLM.
We prove Set-LLM guarantees robustness to permutations, eliminating order sensitivity.
We demonstrate that Set-LLM can be trained effectively with different base LLMs, consis- tently matching or outperforming the base models with random-order inputs, and signifi- cantly outperforming them with adversarial
inputs.
This enables more robust and efficient evaluation framework for multiple-choice question answering and for the use of LLMs in multi-document applications.
2
Background: transformers and positional encoding
The Set-LLM adaptations involve changes to the attention mask and positional encoding of transformer-based LLMs. We first describe these components before introducing Set-LLM.
2.1
Attention scores
Most state-of-the-art L.LMs are based on the transformer architecture, made up of multiple attention layers stacked on top of one another 43) Given the d-dimensional representation X E Nxd of N tokens, raw (unnormalized) attention scores (or weights) are calculated as
Z attn X, X, Wo, WK, dx) = XWg(XWx)* /V
(1)
where Wo, WK E dxd, the query and weight matrices, respectively, and dx is scaling factor often chosen to the dimension of the keys.
In causal transformer, attention scores are masked to ensure that tokens can only attend to preceding tokens, before being normalized through softmax layer. Masking is usually represented by matrix of 1's and 0's, . E {0, NxN, where M3; = 1 if token i can attend to token j. However, masking can also be denoted by directed graph GM = (V, E'), where (j.i) E E(GM) + M,, 1, i.c., if information flows from token j to token i. So for a causal mask, (j. E E(GM) if and only if j < Let Ar {j (j.i) E(G')) denote the neighborhood (or field of view) of the token i.
Then we can write the normalized attention weights as
exp( 2ig) As softmax (Zz) ELEN, exp (Zi+)
exp
(2)
Ee

---

if (j.1) E E(GM). and A.₃ = 0 otherwise.
LLMs all use some kind of masking. Decoder-only architectures causal attention mask, whereas encoder-decoder architectures, such as T5 (4, used bidirectional (or fully-connected) attention for the prefix (or prompt) and causal attention for the output (or response). We refer this as prefix masking. Figure iillusraate the causal and prefix masks in both the matrix and graph forms.
Given normalized attention weights, the attention layer is completed by taking weighted averages of the token neighborhoods:
x = A x000
(3)
where A7 is the value matrix, and the superscript indicates the model layer.
2.2 Positional encoding
In addition to the causal mask, transformers use positional encoding to introduce positional infor- mation. There are common variants: absolute positional encodings and relative positional encodings. Absolute positional encodings assign consecutive integers to the tokens, starting at O. These are usually embedded with encoder layer and concatenated with the corresponding input token embeddings. The neural network can use these embeddings to "understand" word order. The attention scores depend on the token embeddings X, and the absolute positions and j:
(4) The positional information is often incorporated into the token embeddings, so row i of X is I; = (3;; i) for some function 1b : T x N- + Rd, and there no further dependence on reducing attn.bs to Equation [ We formulate our proofs using this notation.
Zij = attn (X,, X,, Wq. WK, i.)..
On the other hand, relative positional encodings use the relative distance (i -6--) j) of tokens in attention calculations. The attention scores depend on the token embeddings X4. Xx,, and the relative distance:
Zij attnyel (X;, x,, WQ: WK. <k, j).
(5)
In this sense, relative position encodings can be seen as a special case of absolute position encodings,
where translation symmetry on positions is enforced, i.e. shifting the absolute positions by does not change the attention scores and therefore the attention layer outputs:
attn(X,, Xy, Wo, WK, dx, i - j) = attn Xg, Wo, WK, dx, (i + m) - (j + m)).
(6)
Methods: Set-LLM
LLMs are used for a variety of tasks, and many of them have sets in the input instructions. This includes answering multiple-choice questions and comparing LLM--gnnrated outputs. For example, [["Which city is the capital of France:"). (" ("Budapest", "Paris", "Heidelberg", "Zurich")].
More generally, if T denotes the token vocabulary, then mixed set-text instruction can be written as -= [go. 41...., qn., where each qi is a set of token sequences: q {s0, {10..1..... $1.... Sn, } and s, = [To, ......, .. with all T; T. If q = [9o] and 14ol = 1, then we are back to the regular case where an instruction is a single sequence of tokens. In the above example, depending on the tokenization, we might have qu = W[hhhic"" city", .I. and qı = (["Buda", "pest"]. ["Paris"].....
For ease of notation, we use global indexing, where tokens in a set of choices are also numbered con- secutively (in the default order in which the set is provided). Then the tokens of are (to,t)... t....,... ttl, with 4 E T, where N is the total number of tokens in q. We use s(4:) to denote the token sequence containing the token and q(4:) denote the containing s(4.).
Since LLMs take sequences as input, one would typically force an order onto sets within mixed input. However, ideally, we would like LLM, whose output does not depend on this order. Our proposed approach, Set-LLM, achieves this in four steps: (1) Remove sequential position encoding, that is, set all positions to 0 (also called NoPE No Positional Encoding); (2) Remove the causal mask and replace it with a prefix mask; (3) Add coomuuation-invariann set position encoding (SetPE); and (4) Add permutation invariint set attention masking (SetMask). The steps are illustrated in Figure
The first two steps already guarantee set permutation invariance. fact, they create a bag-of-words (BoW) model that ignores the order of all input tokens, not jout the order of elements within a set. Clearly, BoW models have their limitations, since word order is critical to language. Steps (3) and (4)
therefore crucial in reintroducing the order information within the different spans of text.
3

---

Prompt Responee
Causal Mask PE
Foational Encoding
Prefix Mask NoPE (BoW model)
4. Add SetMask
3. Adi SHIPE
Set Mask SetPE
(Set-LLM)
Three types of attention masks and their corresponding directed graphs. The colored squares on the left indicate attention scores that are not masked. For example, in a causal mask, the 4th token attends to the first 4 tokens, and the remaining tokens are masked. The circles on the right represent the tokens as nodes an attention graph. Red, orange, and yellow tokens correspond to the prompt, and blue tokens correspond to the response. Orange and yellow tokens corresponds elements of a set. Causal masks are standard in decoder-only LMs, whereas prefix masks are used in bidirectional encoder-decoder LMs. SetMask is introduced in this work. Some edges are grayed out and self-loops are omitted to improve clarity. In addition, the figure shows three types of token positions, standard consecutive positions (PE), no positional encoding (NoPE), and set position encoding (SetPE). These are indicated by the numbers inside the token nodes on the right.
Figure 2:
Where is the Eiffel Tower?
(Heidelberg
Zurich
Paris
(Budapest
Please answer...
Figure 3: An example of a multiple-choice question with set positional encoding (SetPE) positions.
3.1 Set position encoding (SetPE)
While BoW models inherently disregard the order of set elements, they also disregard the word order within the text. To overcome this limitation, we introduce set position encoding (SetPE). For SetPE, generate SetPE positions and use them calculate absolute relative positional encodings. The idea to number tokens consecutively, but to number elements set from the same starting position. An example is provided in Figure along with pseudocode in Section B
The example shows how SetPE positions align with standard absolute positions for "regular" text (starting at 0). However, when set of options appears (e.g., at position 6), all options within that set are numbered consecutively from that starting position. This ensures that no order forced upon the options, but the token order within the options is clear. Positions resume with their absolute positions after a set of options (continuing with 12 in the example). We denote SetPE positions by the function (or pos in the pseudocode), i.c., the position of the 1th token of query q is written Φ(9)1; pos[i].
Given the SetPE positions, we can calculate absolute or relative positional embeddings. When using absolute positional encoding, the SetPE positions simply replace the absolute positions and are encoded and concatenated with the input token embeddings. When using relative positional encoding, the difference between SetPE positions is used to calculate relative positional embeddings rather than the difference between absolute positions. All LLMSs this paper specifically use RoPE [39]. this case, the SetPE positions determine the angles of rotation for the token embeddings.

---

two facts about
Hore about
think great
think Paris
think London is awful
think London
10
Which city do I prefer?
Which city do prefer?
Tokens and their positions
Answer: Paris
Answer: Paris
Figure 4: A failure case for an LLM with set position encoding (SctPE) but without set attention mask (SetMask). SetPE positions are shown for the tokens that are part the set (of facts). Without the SetMask, the model is unable to distinguish the two inputs. The model can not "know" which position token belongs to which position 10 token.
3.2 Set attention mask (SetMask)
When using prefix mask, SetPE is insufficient for distinguishing mixed inputs. Consider the example of two inputs in Figure which contain sets of opposing facts. Clearly, one person prefers Paris, while the other prefers London, but an LLM with prefix masking and SetPE will output the same next token distributions. To see this, note that all tokens in the cases receive the same SetPE positions, although greal and awful have been switched. Since the prefix mask is fully connected, the attention layer outputs are then identical up to switching the two respective embeddings.
To address this shortcoming, we introduce set attention masking (SetMask). The idea is to use the attention mask distinguish between the two cases in Figure 4. SetMask is constructed by starting with prefix mask and removing all edges between tokens of different elements of the same set (that is, setting the respective matrix entries 0). An illustration in both matrix and graph forms can be seen in Figure 2 (bottom). More precisely,
0 if (t+) = ((g)) and stt)) + s(t,)
M1 =
1 else.
Since generation is autoregressive, response tokens can only attend to prompt tokens and preceding re- sponse tokens. SetMask is therefore extended to the response in the same way causal mask and pre- fix mask. With SetMask, the tokens corresponding to great and awful in Figure havve different neigh- borhoods in the two inputs, leading to different embeddings and different next token distributions.
3.3 Permutation invariance
We claim that by construction, an attention layer with SctPE and SetMask set permutation equivari- ant. Then, since all intermediate layers set permutation equivariant and the final layer of an LLM is permutation invariant, follows that the whole network is set permutation invariant [4]
Theorem 1 (Set Permutation Equivariance). Let π be a permutation corresponding to permuting
elements of sets mixed sel-text input, and let be the corresponding permutation matrix. X(++1; A Wy is the output of an attention layer with SerPE, SetMask, and absolute positional encoding, then ((++1) = P (141), In other words, attention with SetPE and SetMask is equivariant to set permutations of mixed set-text input.
Theorem 2. Attention with SetPE and SetMask reduces to attention with PE and prefix masking when the input is a single sequence of tokens (ie., when the input does not contain sets).
Both proof' can be found in Section
4 Experimental setup
To evaluate Set-L.LMM we test with models on multiple-choice datasets. Multiple-choice ques- tions are the ideal testbed for Set-LLM since there widely-available, well-established benchmarks.
5

---

Datasets. Models are evaluated on four popular multiple-choice reasoning benchmarks: PIQA B ARC-Challenge [3]. Commonsense [40], and SIQA [3]. Each task consists of a series of questions, each with multiple choices, where only one answer is correct. The number of choices per question varies by the dataset. the original setup, PIQA, ARC-Challenge, and CommonsenseQA provide only the question in the prompt, and the different answer choices are run through the model to test them as continuations. The choice with the highest log-likelihood is selected as the answer.
To adapt these datasets to our research setting, we modify the prompt so that the choices are provided as part of the question, and the order of these choices can be permuted. Finally, in the case of SIQA, although the original prompt includes the choices, we remove the numbering since this implicitly assigns an order to the options and breaks the permutation invariance. All original and modified prompts can be found in Section D.3
We use the standard train-evaluatto splits for all benchmarks.
Additional pretraining data. The Set-LLM adaptations fundamentally change the input of the model and the inner attention mechanism, so the adapted models require training to function in their new setups. To help the models, we experiment with additional pretraining. We use a high-quality subset (approximately 10k examples) of the cleaned Ultrafeedbcck instruction-following dataset (7]. attained by following the data preprocessing steps in [2T] We use Ulra in the results to indicate additional pretraining, More information on instruction instruction-finetuning can be found in Section D.I
The experiments involve two evaluation modes: (1) Random Order: For each input question, we test al al permutations of the answer choices, and calculate the average accuracy, (2) Adversarial Order: For each input question, we test all permutations of the answer choices and use a permutation where the LLM returns a wrong answer, if one exists. Accuracy is then the proportion of questions for which the LLM remains correct across all possible permutations.
Evaluation modes.
Base models. We evaluate the proposed method using several popular pretrained decoder-only language models: Gemma 2B and 7B [42. Llama 3.2 1B, Llama 3.2 3B, and Llama 3.1 8B We select these models to test different architectures and model sizes.
Baselines. We consider the following baselines: (1&2) A pretrained and a finetuned LM using the original dataset prompts (Causal Mask+PE" Pretrained & Causal Mask+PE Finetuned), (3&4) A pretrained and finetuned LM using the modified dataset prompts (Causal Mask+PE Pretrained & Causal Mask+PE Finetuned), (5) A finetuned LM with additional pretraining on UltraFeedback ? using the modified dataset prompts (Causal Mask+PEUln Finetuned).
An alternative to a permutation- innrariationnnvaiiaatt invariant architecture is run all possible permutations of the input through the LLM and pick the option with the most "votes" [ make any model overhead, since the model has to be run k! times, where A is the number of options. We include majority vote for baselines 3, 4, and from above. These models use the modified prompt containing the answer choices, which can be permuted to calculate the majority vote. Finally, include another permutation-invant
This approach, majority vote [57] can However, it also comes with an exponential factor runtime
cormuutationiivariiant
baseline: no options. Here, the questions are asked without providing the answer options. We finetune Causal Mask+PEUlle model each dataset.
Training setup. We update the model weights using LoRA [5] applied to all linear layers of the multilayer perceptron (MLP) and self-attention layers. Details about the hyperparameter settings can be found Section D.4 We finetune models separately on each benchmark. We train all models on a single Nvidia H200 GPU with training times ranging between 1 and 4 hours for one model on one benchmark. A comparison of baseline and Set-LLM runtimes is provided in Section
E7
We train all our models with 6 precision. However, we use full 32-bit floating point precision for all evaluation runs. This proves crucial in ensuring permutation invariance in practice. Although the Set-LLM architecture is provably can lead to a different order of the low-level computations resulting in minor inconsistencies. which add up layer by layer. We do not observe any inconsistencies using 32-bit Moating-point precision.
corimutation-ivvaiii (Theorem T]. permuting the input tokens
For CommonsenseQA. only the first 24 (of a possible 5! = 120) permutations are tested. "google/gemma-25 google/gemma-72
meta- -aaaaaaamaaaa..--- 1.--stttutt neta--lamaa/ Instruct, meta- aetta-llama/l..o 1ama--.1-88--nme [46] Huggingface]
In the event of tie, the winner is chosen uniformly at random from the top-v options.

---

Table 1: Gemma 2B baselines on four multiple-choice datasets. All scores are accuracies (%). "Results using the original dataset prompts, which for PIQA, ARC, and CSQA only contain the question. All other results use modified prompts with choices provided as part of the question.
Model
Training
Eval. Mode
PIQA Adv.
ARC
CSQA
SIQA Rand. Adv.
Rand.
Rand. Adv.
Rand. Adv.!
Random
50.00
50.00
25.00
25.00
20.00
20.00
33.33 33.33
Causal Mask+PE'
Pretrained Finetuned Finetuned No Options
Single Single run
76.77
37.80
51.76
37.26
Causal Mask+PE"
79.82
45.39
68.80
75.95
Causal P
79.87
43.26
69.37
56.55
Causal Mask+PE
Pretrained
Single run
57,45
30.96
36.03
7.68
34.92
16.46
39.29
12.74
Causal Mask+PE
Pretrained
Majority Vote
56.04
56.04
40.10
40.10
35.22
35.22
40.23
40.23
Causal Mask+PE
Finetuned
Single run
84.11
76.77
55.20
23.72
78.31
69.62
74.80
63.00
Causal Mask+PE
Finetuned
Majority Vote
84.06
84.06
58.87
58.87
78.38
78.38
76.05
76.05
Causal PPl
Finetuned
Single run
83.98
77.31
56.32
26.88
77.89
68.47
74.33
63.97
Causal Mask+PE
Finetuned
Majority Vote
83.57 83.57
59.56
59.56
78.46
78.46
75.23 75.23
"Results with original prompts
Ulaa Additional pretraining
TOnly first 24 (of 120) permutations tested
Experiments and results
5.1 Baselines and order sensitivity
We first run baseline models and measure the gap between random-orde order and adversarial accuracies. We use Gemma 2B the base LLM in the first experiments.
T the results for all Gemma 2B baseline models. Finetuning Gemma 2B on the datasets (Causal Mask+PE Finetuned) gives competitive results in the random evaluation mode. However, adversarial permutations lead to large accuracy drops, particularly for ARC-Challenge (55.20% to 23.72%). In contrast, the majority vote baselines do not have any drops in accuracy between the two evaluation modes, confirming that they are connuutatoo-ivvariinn best random-order results. However, this comes a high cost, as the models have to be run k! times for each input. We use Causal Mask+Pr Finetuned as the baseline for further experiments.
Moreover, they also produce the
5.2 Set-LLM step-by-step
As described in Section 3, there are four steps in turning a base LLM into a Set-LLM. To gain better understanding of the individual steps, Causal NasK+NoPE, Prefix Mask+NoPE. Prefix Mask+SerPE, and SerMask+ (Set-LLM). Finally, include Prefix Mask PE in experiments, which is encoder-decoder version of the base LLM. Note that in addition to Set-LLM, Prefix Mask+NoPE and Prefix Mask+SetPE are also set-permutatioon-nnvaiinn models and therefore perform exactly the same in the modes.
experiments with the intermediate models:
Table shows the results for all intermediate models. The cet-permutaiionnivaaric have any drops in accuracy in the adversarial setting, confirming design choices and theoretical results. Moreover, Set-LLM outperforms the strongest baseline on all four benchmarks both modes, indicating that the permutation -nvariianceee guarantees do not come at cost to accuracy. This is all the more impressive considering these results come from a single run, rather than runs. Prefix Mesk+SetPE is not far off SetMa but SetMask is needed to outperform the majority vote baseline and give the best results. The problem illustrated in Figure might help to explain this gap. Additional analyses of the majority vote versus Set- LLM output can be found in Section E.8
models do not
5.3 Different base LLMs
In addition to Gemma 2B, we evaluate Set-LLM using Gemma 7B, Llama 3.2 1B, Llama 3.2 3B, and Llama 3.1 8B as base models. drops between the two evaluation modes ranging from 3.4% to 31. 31.7%. In contrast, there are no drops with Set-LLM, and Set-LLM outperforms the base model 20/20 cases with adversarial ordering and in 18/20 cases with random ordering. Moreover, with single run, it outperforms majority vote
that base models suffer from order sensitivity with
7

---

Table 2: Set-LLM and all intermediate models going from the base model (Causal Mask+PE) to
Set-LLM (eet.a The adaptation steps are described in Section 3 The base model is Gemma 2B. All models are finetuned separately for each benchmark. All scores are accuracies (%).
Steps
Model
Eval. Mode
Runs
PIQA
ARC
CSQA Rand. Adv.
SIQA Rand. Adv.
Rand.
Adv.
Rand. Adv.
Causal Mask+PB fltra
Majority Vote
kd
83.57
83.57
59.56
59.56
78.46 78.46
75.23 75.23
Causal Mask+PE
Single run
84.11
76.77
55.20
23.72
78.31
69.62
74.80
63.00
Causal Nask+NoPE
Single run
1
74.37
63.55
35.70
14.76
68.49
57.33
63.21 48.57
1,2
Prefix Mask+NoPE
Single run
I
74.54
74.54
32.08
32.08
49.14
49.14
51.02 51.02
2
Prefix Mask+PE
Single run
I
82.78
76.50
57.62
27.47
78.98
71.01
74.36 65.66
1-3
Prefix SePK+
Single run
81.23
81.23
51.28
51.28
77.31
77.31
71.24 71.24
1-4
S
Single run
1
84.33
84.33
57.76
57.76
79.93
79.93
75.38 75.38
1-4
Set
Single run
1
85.80
85.80
65.02
65.02
80.18 80.18
76.15 76.15
Additional pretraining
OOlly first (of 120) permutations tested
k: number of (multiple) choices
Table Set-LLM performance with different base LLMs. All models were pretrained on UltraFeed- back 7 and then finetuned separately for each benchmark. All scores are accuracies (%).
LLM
Model
Eval. Mode
PIQA
ARC
CSQA Rand. Adv.
SIQA
Rand.
Adv.
Rand. Adv.
Rand. Adv.
Causal Mask+PE Ulta
Single run
83.98 77.31
56.32 26.88
77.89 68.47
74.33 63.97
Gemma 2B
Causal PPsk+
Majority Vote
84.17 84.17
60.15
60.15
78.71
78.71
75.38
75.38
SctMask+SetPE
Single run
85.80 85.80
65.02
65.02
80.18
80.18
76.15 76.15
Causal Mask+Pp.
Single run
92.82 89.45
83.52
64.33
85.45
79.12
80.93
74.10
Gemma 7B
Causal Mask+PpUUle
Majority Vote
92.66
92.66
85.58
85.58
85.75
85.75
81.10 81.10
SetMask-SSrPP:
Single run
92.98 92.98
83.45
83.45
84.93
84.93
81.12 81.12
Causal Mask+PPUue
Single run
79.57 71.33
53.61
21.93
74.50
64.21
71.84 62.79
Llama 3.2 1B
Causal Mas
Majority Vote
79.49
79.49
57.17
57.17
75.51
75.51
71.85 71.85
SetMask+SSiPP
Single run
81.66 81.66
59.30
59.30
76.66
76.66
72.47 72.47
Causal Mask Maek+
Single
86.92
81.72
74.16
53.07
81.32
74.94
77.54 70.42
Llama
Causal Mask+PE. Pr
Majority Vote
86.83
86.83
76.37
76.37
81.57
81.57
77.99
77.99
SeiPP
Single
88.41 88.41
75.85
75.85
83.29
83.29
80.30 80.30
Causal Mask+PpUlte
Single run
90.81 86.29
83.04
64.51
83.96
77.89
80.77
73.90
Llama 3.1 8B
Causal Mask+PEUlie
Majority Vote
90.75 90.75
85.32
85.32
84.11
84.11
81.12 81.12
Se
Single run
91.62 91.62 84.13 84.13
85.34
85.34
81.47 81.47
Ulsa Additional pretraining
'Only first 24 (of 120) permutations tested
in 16/20 cases, without the exponential runtime overhead. Set-LLM outperforms the baselines across all model architectures and sizes we tested. Additional results can be found in Sections F.3 and E.4
5.4 Out-of-distribution performance
When using Set-LLM as an LLM evaluator, is particularly important that the adaptations do not hurt the performance on out-of- nut-of-distribution data after finetuning on a small dataset. We measure the out-of-distribution multiple-choice performance of the models by finetuning model on each benchmark independently and then evaluating on the three remaining benchmarks. We do this for both the base model and Set-LLM and compare their performance with the pretrained base model. Results are provided in Table 18 in the Appendix. Set-L.LM is the best-performing model in 10/12 cases in both evaluation modes. We also measure the impact of the Set-L.LM adaptations on standard (non-set) language task performance. Please refer to Section E.6 for these experiments.

---

Table 4: Multi-document summarization with different base LLMs. All models were pretrained on UltraFeed-bace -ack 7 and then finetuned on MultiNews 9..
All Rouge metrics are F1 scores.
LLM
Model
Compression
Rouge-1
Rouge-2
Rouge-1"
Rouge-2"
Target Summaries
6.09
0.27
0.14
Gemma 2B
Causal Mask+PEUlln
15.84
0.17
0.12
0.33
0.11
SetMask+SePPEE
7.23
0.27
0.20
0.47
0.19
Llama 3.2 IB
Causal Mask+PDulm
20.76
0.13
0.08
0.33
0.10
SeMMaak
7.31
0.27
0.18
0.48
0.18
Llama 3.2 3B
Causal P
9.77
0.23
0.15
0.43
0.15
SetMask
7.23
0.27
0.18
0.49
0.20
'Rouge Fl scores between the target summaries and the model summaries.
6 Potential use cases
We include additional proof-of-concepp experiments to demonstrate the benefit of Set-LLM in three different LLM applications: multi-document summarization, multi-document question answering, and L-L-aas-a-judee (Section E. We see promising results all three use cases.
6.1 Multi-document summarization
In multi-document summarization the LLM is presented with a set of documents and tasked with producing collective summary. We perform our evaluation on the MultiNews dataset (9. Details about the experimental setup can be found in Section D.5
We report standard summarization metrics, namely n-gram Rouge Fl scores between the model summaries and the input documents and compression rate the length of the original text divided by the length of the summary. We also include Rouge F1 scores between the target summaries and the model summaries Rouge-1* and Rouge-2". The results are summarized in Table Set- outperforms the finetuned baseline with all three base models in all metrics. Note that Set-LLM produces longer summaries, which leads to lower compression rates.
6.2 Multi-document question answering
80
Motivated by 27], evaluate Set-LLM in swering multi-document questions. The task consists of questions with 20 supporting docu- ments, where only one document contains the information required for the answer. The authors show that many LLMs (including commercial models) produce a U-shaped performance curve with respect to the location of the answer. If the relevant document listed first last, then the LLM correctly answers the question with higher probability. In contrast, Set-LLM guar- anteed to have flat curve, since reordering the documents does not affect the output. We test five different locations (Ist, 5th, 10th, 15th, and last) for the placement the answer document. We provide results for two LLama-based models and compare these with GPT-3.5-Tune scores from [27]. Details about the experimental setup can be found in Section D.6
Accuracy
LLama 3.2. JB, Causal Musk Piehtau
1B, Causal P
50
LLama 3.2 IB, SetMask+SerPPling
6PT-3.5-Tubbo
27
40
10
15
20
Position Document with
Figure 5: Varying the position of the relevant docu- ment (containing the answer) within the pretrained model's input results in a U-shaped performance curve. However, Set-LLM (or finetuning) produces flat performance curves with a higher accuracy. 1GPT-3.5-Tubbe was not finetuned on the dataset.
As expected, the results in Figure 5 demonstrate that Set-LLM is not sensitive to the location of the relevant document. Furthermore, Met-LLM outperforms the baseline model, even after finetuning. The pretrained base model produces a U-shaped curve in line with previous work [27), but the

---

finetuned version produces a mostly flat curve. For larger models, the advantage of Let over the finetuned base model seems diminish. For results with Llama 3.2 3B, refer to Section E.9
7
Related work
To the best of our knowledge, this is the first paper to specifically introduce a permutation-invarian architecture for decoder-only language models. Concurrently to our work. Kinder et al. [19] propose Set-Encoding to address positional bias in LLMs. Similar to work they propose to modify the positional encoding and attention mask for set-valued input.
Permuting multiple-choice questions. This work was motivated by the observations that you can "fool your (vision and) language model with embarrassingly simple permutations" [57]. The authors quantify the effect of adversarial permutations also analyze the effectiveness of majority vote, which we include as a baseline in our experiments. Prior to this, Liu et al. [29] suggested rotating the choices of multiple-choice questions evaluate the robustness of (Multimodal) Language Models. Taking a majority vote over the rotations could be used alternative to majority vote, thereby only carrying linear-factor rather than an exponential- factor runtime overhead, but is not guaranteed to give the same answer for all permutations.
(V)LLMs and multiple-choice benchmarks. They
LLM evaluators. LLM evaluators (or judges) were introduced recently [3 153, but have already been integrated in many LLM pipelines, including for evaluation, retrieval and reasoning 2 TO [2] 26). The order bias of LLM evaluators was already observed by Zheng et al. (53. and methods involving the aggregation of multiple runs and specialized prompting have been proposed to mitigate this problem [18 29 4T 56 (7). However, this is the first work to eliminate order bias directly from the model architecture without impacting accuracy or runtime.
Graphs and large language models. A set be seen an empty (or fully-connected) graph, and As a result, several works that 48 49) Most closely related to this work are the papers of Liu et al. (28) and Plenz and Frank [33], who adapt an encoder-decoder and an encoder-only language model, respectively. to take mixed graph-text data input. Both approaches alter the attention mechanism and the positional encoding to incorporate graph connectivity information into the embeddings.
indeed one of the key features of graphs is also permutation-invarianee
combine graphs and LLMSs are also relevant here [14
Invariant neural networks. While invariant networks have existed for some time, Bronstein et al. 4 recently unified them under a common geometrie framework. For example, CNNs [23 52 are shift-invariant architectures for images, Deep Sets (50] was the first cormutationninvariin architecture are cormuuaaionninvariann architectures for graphs. In this work, we combine the permutation invariance of these architectures with the power of large language models.
for sets, and GNNs 20 44
8 Conclusion
To the best of our knowledge, this paper introduces the first permutation-invaare decoder-only LLM, Set-LLM. We formally prove that Set-LLM is permutation-invarian and show how robust it is to permutations in practice. The models are based on pretrained LLMs, can finetuned efficiently with low-rank adapters, and incur additional runtime costs. We test Set-LLM four multiple-choiee datasets, where it outperforms all baselines by significant margins in either accuracy or runtime. Moreover, we show promising results using Set-LLM in two important applications: multi-document summarization and multi-document question answering.
With the growing importance of multi-docunan applications, Set-LIM has the potential for wide impact. We provide promising initial experiments for natural use cases, and we hope that future work can explore these directions in detail.
Limitations. While we propose a general purpose pnermuutaiooniivarian LLM, the approach requires explicit knowledge of which parts of the input are set-valued, which may not be readily available in some applications. There no obvious way to combine Set Mask with sparse attention mechanisms, e.g. sliding window attention. This would be an interesting direction for future work.
10

---

Acknowledgments
The authors would like thank the Vector Stiftung for their financial support in the framework of the MINT Innovations program.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.085 08774, 2023.
[2] Sher Badshah and Hassan Sajjad. Reference-guidd verdict: Lims as-jdges in automatic evaluation of free-form text. arXiv preprint wrXi:2248.009235 2024.
[3] Yonatan Bisk, Rowan Zellers, Gao, and Y Choi. Piqa: reasoning about physical commonsense in natural language. corr, vol. abs/1911.1164A) 2019.
[4] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2 13478, 2021.
[5] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. vrXiv:/803.054555 2018
[6] Marta R Costa-Jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi. Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centern machine translation. arXiv preprint arXiv:2207.04672 2022.
[7] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting lan- guage models with scaled feedback, 2024. BtLl:htps:///att
org/ abs/a/// 01377
[8] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30069, 2023.
[9] Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv: 01749,, 2019.
[10] Chang Gao, Haiyun Jiang, Deng Cai, Shuming Shi, and Wai Lam. StrategylIm: Large language models strategy generators, executors. optimizers, and evaluators for problem solving. Advances in Neural Information Processing Systems, 77796797-96846. 2024.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
[12] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
[13] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversar- ial attacks against text transformers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings the 2021 Conference in Natural Language Processing, pages 5747-5757, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational L Linguistics. doi: 10. 10.18653/v1/202/ emnlp-main.45 464. URL httpt:t:/tccaaaaitoo//// wrw.llooo.oor//g// emnlp-main. 464/
Empirical Methods
[14] Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: Retrievallaugmenned generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876-132900 2024.
11

---

[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL htttps::////pereeee net//orrum?ddnnZaWWeeree
[16] Albert Q Jiang, A Sablayrolles, Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b, arxiv. arXiv preprint arXiv:2310.06825, 10, 2023.
[17] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892 24928, 2023.
[18] Seungone Kim. Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim. Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat. Jinheon Back, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5877-5919, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN org/2025 nww.laacclaaaa long. 3033
179-8-891766 19177-189-66
[19] Lukas Kinder, Lukas Edman, Alexander Fraser, and Tobias Käfer. Positional overload: Posi- tional debiasing and context window extension for large language models using set encoding. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1896-3908 Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 11199-889176-222 -O. doi: 10.18653/v1/2025 long.ong..19 URL
org 2025 acl long. 197/
[20] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXi:: 02907, 2016.
[21] Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Bitune: Bidirectional instruction- tuning. arXiv preprint arXiv:2405. /4862, 2024.
[22] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[23] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
[24] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šažko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing, In Heike Adel and Shuming Shi. editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi: 10. 10.118633/1/2221.coom http/22211..mmpdddmmm.mm.. URL halLitctss///cllatooooogg org 2021. http:/2221eemmmddeeeeee momlp--dmmo.. 21/
[25] Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on in finance, pages 374-382, 2023.
12

---

[26] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujuu Yang. Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889-17900 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18611/663vv1 hetp:024emmlp-mmaae 992. URL httpp:://acaanthllllla org/ 2024. wwwnl-mmmaiii m91..992n/99
[27] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03112, 2023.
[28] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2901-2908, 2020.
[29] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[30] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the- purameter-efficien fine-tuning methods. https
//githu.. wwwthuu..om/uugggiii http://hggnnnfccccc peet 2022.
[31] Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan, Xinyan Hu, Wei-Lin Chiang, Anastasios N Angelopoulos, Trevor Darrell, Narges Norouzi, et al. Search arena: Analyzing sarch-augmented llms. arXiv preprint arXiv:2506.08 05334, 2025.
[32] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[33] Moritz Plenz and Anette Frank. Graph language models. In Lun-Wes Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4477-4494, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10. 12.18.53//// 245.
URL UBLL/ttppe///ccaaaaollo.o//////p wwwp:/y...rg.ooo2..ggg http:/2224.allloogg long. /45/
[34] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(14)):1667, 2020.
[35] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't Gon't know: Unanswerable questions for squad. arXiv preprint 1806::8800033222 2018
[36] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 4463-4473, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/DD1911454 URL 1454/
[37] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- fics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: (0.18653/v1NN1--22
URL htsl:ttta///allanoollogg hrttcchoooggoooog/// org//18 NN8-2074/
[38] Erfan Shayegani, Md Abdullah AI Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu- Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks.
arXiv preprint 2xX1v:2310.10844 2023.
13

---

[39] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Neurocomputiing, 568:127063, 2024.
[40]
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4149-4158 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//119-42211 URL https://
aclanthology.org wrg. N111//
[41] Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. Found in the middle: Permutation self-consistency improves listwise ranking in large language models, arXiv preprint arXiv:2310.07712, 2023.
[42] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson. Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Prnquette-Choo Clément Crepy. Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christooe Christian Muraru, Grigory Rozhdestvenskiy. Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri. Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024. URL httpttt///xxxxi..... org. abs/2403. 08295
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1xXi::7700.10000 2017.
[45] Peiyi Wang. Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong. Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-We Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 9440- 9450, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v 1 aow.o.... URL https: https:://ccclaaa haap:::/aalaatthhloggoo org/ 22gg/2024..o .aoo24.ccl--1 acl- 1oll-oong.511 511/
[46] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv: 1
[47] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: (ttv::881..000 1810.00826, 2018
[48] Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34:28798- 28810. 1021.
14

---

[49] Minji Yoon, Jing Yu Koh, Bryan Hooi, and Russ Salakhutdinov. Multimodal graph learning for generative tasks. In NeurlPS 2023 Workshop: New Frontiers in Graph Learning, 2023.
[50] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[51] Shengyu Zhang. Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey. arXiv preprini arXiv:2308. 10792 2023.
[52] Wei Zhang, Jun Tanida, Kazuyoshi Itoh, and Yoshiki Ichioka. Shift-invariant pattern recognition neural network and its optical architecture. In Proceedings of annual conference of the Japan Society of Applied Physics, volume 564. Montreal, CA, 1988.
[53]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 00:46599-46623, 2023.
[54] Yanxin Zheng. Wensheng Gan, Zefeng Chen, Zhenlian Qi. Qian Liang, and Philip S Yu. Large language models for medicine: a survey, International Journal of Machine Learning and Cybermetics, 16(2):1015-1044 2025.
[55] Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint 1XX::2310.17631 2023.
[56] Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. A setwise approach for effective and highly efficient zero-shot ranking with large language models. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 38-47, 2024.
[57] Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. In Forty-first International Conference on Machine Learning, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The theoretical contributions (Theorems | and 2 are accompanied by proofs
in Section C and the other claims are supported by the experiments and results in Section S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: There is a separate paragraph describing the limitations in the conclusion (Section R).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: Full proofs of (Theorems] | [add] are provided in Section C Assumptions are included in the theorem statements.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The code will be made available in a public repository upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be made available in public repository upon publication. We include hyperparameter values in Section D.4 the supplementary material and will also include these along with scripts to run specific experiments in the code release.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
//nip. ptaips.cc/public/uiddee http//udddee//eeummmii seriiisssonpoiiiio
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The code will made available in a public repository upon publication. We include hyperparameter values in Section D.4 of the supplementary material and will also include these along with scripts to run specific experiments in the code release.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because would be too computationally expensive. Instead, focus was placed on including additional base model architectures. Considering the consistency results across different base model architectures and different benchmark datasets, we believe the paper's claims strongly supported.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Section E.7
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:pppp//aaaaee..cccc/cpahaaaa
Answer: [Yes]
Justification: The authors have read and acknowledged the NeurIPS Code of Ethics. The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
19

---

Justification:
There is paragraph in the conclusion (Section 8 dedicated to potential
impacts. We have also included section dedicated impact in the supplementary materials to give further details about societal impacts (Section Aj.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We not do release any new pretrained language models. All of the models and datasets used in this paper are already freely and readily available the public domain.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of these details are available in Section D of the supplementary materials or mentioned in the experimental setup section the main paper (Section
Guidelines:
The answer NA means that the paper does not use existing assets.
20

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaeed, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All model details, training setups, and hyperparameter values are included in the paper. The code will also made publicly available upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standaadd components.
Please refer to our LLM policy for what should or should not be described.
22

---

A
Impact
We do not foresee any direct negative impacts from this work. On the contrary, we believe this work, as with any other work on model robustness, can contribute positively to LLM applications, especially in high-risk scenarios. Moreover, this work can used develop more robust evaluation approaches, which can help move the wider field forward.
B
SetPE algorithm
Algorithm 1 Python-like pseudocode for calculating Set Position Encoding (SetPE) positions.
input mixed list of token ids and aeta of token id lists
# vocab token vocabulary
pos [] list of SetPE positions
ind running position index
for q in input: # iterate through elements of the input
if q in vocab: if element is a token
aes.apeddd
ind += 1
else:
element q set of token lists for s in # itteraee through set of token lists # append consecutive positions for each token in 8: pos pos list (range (ind, ind+len(s))) total_tokens.p sum (s) for q]) ind total_tokens__
return pos
C Proofs
Theorem 1 (Set Permutation Equivariance). Let π be a permutation corresponding to permuting elements of sets mixed set-text input, and let be the corresponding permutation matrix. X(++1 A Wy is the output of an attention layer with SerPE, SetMask, and absolute positional encoding, then ((+1)) = P (X41), In other words, attention with SetPE and SetMask is equivariant set permutations of mixed set-text input.
Proof of Theorem I.
Claim 2.1. PX
First note, that by the definition of SctPE, SetPE positions will be the same before and after the set permutation is applied to the input. This is because all token sequences remain in the same set after set permutation, and all token sequences within set numbered consecutively from the same starting index. If denotes variables after permuting the input, then āz(s) (b(t, pos(t]) = .q Tokens that are not part of set are not moved by set permutation, क T(i) = for these tokens and again x+(i) = & pos(i]) 24. Therefore, X PX.
Claim 2.2. 2 PZP2
By Claim 2.1 have
alin(x, X, K.W Wk,
altn(PX,PM)) Wo, WK,
<PXWQ(PXWK) /Vdx
=PXWOWXX" PT /
=P(XWoWXX" /Vdx) pT PPP?
23

---

Claim 2.3. M PMP
Though this is not the case for general permutations of the tokens, we show that this is the case when is restricted to permuting elements within the sets 9i-
The claim is equivalent to showing that Ma((m))) = M, for all i, This can be seen from the construction of SetMask and the definition of æ. But be more precise, split the claim into multiple cases:
Recall that s(t,) denotes the token sequence containing the token t; and ((t;) denotes the set containing st+,).
Case 1: (t,,) # q(tz), g(t,) * q(t,) ((+x()) * ((v(()) since does not move tokens between sets. Then by definition SetMask, Mz(()(()) Mij.
Case 2: q(t,) = q(t)), s(t)). Since æ permutes whole sequences within a together, all tokens within a sequence remain the same sequence after permutation. Therefore, Ma(D)()) = = MMy-
Case ((t.) = q(tz), # s(t)). Again, since æ permutes whole sequences within a set together, tokens in different sequences of the same set remain in different sequences of the same set after permutation. Therefore, Ma(D)())) 0 Mij.
Claim 2.4. -= PAPT
Again, this claim is equivalent to showing that (((()(()) = Aig for all i,j. Putting the previous claims together, have:
Case 1: Mij = 0. By Claim
2.3
if Mij = 0, then Mn(()())) 0 and Ag Am(on()) 0.
Case 2: Mij
By Claim
2.3 if Mij = then M.(() 1, and we have
expl (x())))
À An(mm)
(8)
Σn M (k) expp (()(()())
Claims 2.2 and
Aij
(9)
Magerp
L
Finally, putting all the claims together, we have:
(1++1) = ( x
(10)
(by Claima X. 2
PA PT PX
(11)
= =
(12)
= PX(+1)
(13)
completing the proof that the attention layer is equivariant.
Theorem 2. Attention with SeIPE and SetMask reduces to atlention with PE and prefix masking when the input is a single sequence of tokens (i.e., when the input does not contain sets).
Proof of Theorem 2 If the input is single sequence of tokens, we have [go], qo = [so]. and so (oo To. 7. In this case, Algorithm assigns consecutive positions for each token the input, that is, (0, 1,2, no..). This identical absolute positions (PE).
Similarly, if there are no sets the input, then SetMask reduces fully connected attention mask on the input, i.c., to the prefix mask.
D

---

D Additional experimental details
D.1 Instruction finetuning
Finally, we introduce Instruction Finetuning as the proposed method relies on this training approach.
Many LLMs use some form of instruction tuning during the (pre- pere-)training process 5T) It involves
a dataset of instruction-answer (or prompt-response) pairs {(g, a) )- The training objective
to maximize the probability of autoregressively generating given q, i.e., to maximize
p(a g)
II 1=1...,a
p(a: 4 ",a
(14)
We use instruction tuning for training all models align to the new position encoding and attention masking setups. Since all probabilities in the product are conditional on q. can encoded without causal mask, i.e., tokens in could attend to earlier tokens in g. This makes instruction tuning ideal for our scenario.
D.2 Dataset details
We all the datasets from HuggingFace Datasets [24] Table 5 provides metadata and Table 6 provides licensing details for each dataset.
Table 5: HuggingFace Datasets path, number of train/evaluation samples, number of choices per question, and number of answer choice permutations for each dataset in this paper.
Dataset
Path
Train. Samples
Eval. Samples
k!
UltraFeedback
openbmb/ Caambmb/U/ItaFeedbaae
PIQA
3
piqa
16113
3084
2
ARC
allenai/ai2
1119
1172
24
CSQA
tau/commonsense qa
9741
1140
120
SIQA
36
social i gqa
33410
1954
6
MultiNews
alexfabbri/multi news
44972
5620
2-10
Natural QA
2
7965
2655
20
2.4c+18
Table 6: Dataset Licenses.
Dataset
License
UltraFeedback 7
MIT license
PIQA 3
unknown
ARC [
CC-BY-SA 4.0
CSQA 40
MIT license
SIQA 36
unknown
MultiNews
9
other
Natural QA
22
27
CC-BY-SA 3.0
25

---

D.3 Prompt templates
Original templates. We provide all the original templates used ereate prompt-response pairs for each dataset, for both training and evaluation.
Table 7: Original prompt templates for each dataset.
Dataset
Original Prompt
Response
UltraFeedback
Question: (instruction)
(answer)<HOS>
Answer:
PIQA 3
Question: {question)
[answer]<BOS>
Answer:
ARC S
Question: {question)
[answer]<BOS>
Answer:
CSQA 40
Question:
(question)
[answer]<HOS>
Answer:
SIQA 36
Given the context, answer correctly the question.
Question:
(/answer_ index <}ESS>>>>>
Context: (context)
Question: (question)
Choices:
(0) [choice0] }
(1) [choicel]
(2) [choice2)
Answer:
26

---

Modified templates. We provide all the modified templates used to create prompt-response pairs for each dataset, for both training and evaluation.
Table Modified prompt templates for each dataset.
Dataset
Modified Prompt
Response
UltraFeedback
Question: (instruction)
[answer<<BOS>
Answer:
PIQA
Question: [question)
aanser <BOS>>
Choices:
(choice0)
[choice1]
Answer:
ARC 5
Question: (question)
[answer]<EOS>
Choices:
[choice0)
[choice |
(choice2) [choice3)
Answer:
CSQA 40
Question: {question) )
(answer) <BOS> <Bnsw
Choices:
[choice0)
(choice1
[choice2)
(choice3)
[choice4)
Answer:
SIQA (36
Given the context, answer correctly the question.
Question:
(answer) <BOS>> <BOS>
Context: (context)
Question: (question)
Choices:
(choice0)
[choicel | (choice2)
Answer:
27

---

D.4 Hyperparameter settings
Table 11: Hyperparameters shared across models and datasets.
Hyperparameter
Value
GPUs
Optimizer
A Linear
LR Scheduler
Weight Decay
0.0
Batch Size
10
Accumulation Steps
10
Warmup Steps
300 (or 10% of update steps)
Update Steps
3000
Random Seed
42
We tnne the learning rate for each model and dataset using logarithmic scale: [1e-4, 3e-4, 1e-3, 3e-3]. The final (best) learning rates are presented in Table 12
Table 12: Learning rates for all final baseline and Set-LLM models on all datasets.
LLM
Model
Ultra.
PIQA
ARC
CSQA
SIQA
MultiNews
Natural QA
Causal Mask+PBUlla
3c-4
le-3
le-3
1c-3
1e-3
1e-3
1c-3
Gemma 2B
SetMask+SedPr.uaa
3e-4
le-3
lc-3
1e-3
1e-3
1e-3
1c-3
Causal Mask+Pp
3e-4
3e-4
3e-4
3e-4
3e-4
3e-4
3e-4
Gemma 7B
SetMask+SetPrE
3e-4
3e-4
3e-4
3e-4
3e-4
3e-4
3e-4
Causal Mask+PE
1e-3
le-3
le-3
1e-3
1e-3
1e-3
1e-3
Llama 3.2 IB
SetMask+SedPEllin
le-3
le-3
le-3
1e-3
le-3
le-3
le-3
Causal Mask+PEUlts SetMask+SetPEUin
3c-4
le-3
le-3
lc-3
lc-3
le-3
Ic-3
Llama 3.2
3o-4
le-3
lc-3
1e-3
1e-3
1e-3
1c-3
Causal Mask+PP
3e-4
3e-4
3c-4
3c-4
3e-4
3e-4
3c-4
Llama 3.1 8B
Stt
le-3
3e-4
3c-4
3c-4
3e-4
3e-4
3e-4
Table 13: We use LoRA 15] to train all our models. We use the HuggingFace PEFT library [30) with default hyperparameter values, unless listed.
Hyperparameter
Value
Rank
Alpha
Target Modules
All linear layers of MLP and Self-Attention
D.5 Multi-document summarization
We perform evaluation on the MultiNews dataset [9] than 20000 characters to satisfy our memory constraints. We use the standard train-validation-tes split, but subsample 1000 test samples. We compare the original model architectures with Set-LLM. performing a single model run in both cases (no adversarial setting). We use the best-performing hyperparameters from the multiple-choie experiments for finetuning.
We filter out inputs of length greater
report standard summarization metrics, namely Rouge scores between the model summaries and the input documents and compression rate (the length of the original text divided by the length of the summary). We also include Rouge scores between the target summaries and the model summaries (R1*, R2*, R3*).
28

---

D.6 Multi-document question answering
We evaluate Set-LLM in multi-document question answering. We use the dataset from based on the natural questions benchmark [22]. The task consists of questions with 20 supporting documents, where only one document contains the information required for the answer. We use the original train-valle iiii-vaiidatoon-esstt split from [22]: 60% training, 10% validation, and 30% test data. We replicate the training data five times and shuffle the documents at random for each input. This done avoid overfitting the model to a specific formulation of each training question. We use the best-performin hyperparameters from the multiple-choice experiments for finetuning. We have five different test sets with the answer document placed in five different locations (Ist, 5th, 10th, 15th, and last). We provide results for two LLama-based models. All scores denote the percentage accuracy.
E
Additional experimental results
E.1
Additional pretraining
To help the models adapt to the architectural changes, we experiment with additional pretraining. We high-quality subset (approximately 10k examples) of the cleaned UltraFeedback dataset 77. attained by following the data preprocessing steps in [21] This additional pretraining aims to help the new, adapted model architectures better adapt the architectural changes. We therefore
hypothesize that adapted models will benefit more from this data than the unaltered baseline models.
We present complete results with and without additional pretraining in Table 14 our hypothesis, the additional data significantly improves the performance of Prefix Mask+PE, Prefix Mask+SetPE, and SetMask+SetPE, especially ARC- ERC-Challenge, but does not improve the performance of the finetuned Causal Mask+PE. However, the pretrained Causal Mask+PE results improve the most, suggesting that the pretrained base model is not particularly well-suited to the benchmark task setups.
Consistent with
Table 14: Results of (ire-)finetuning Gemma 2B on UltraFeedback with the respective attention mask and positional encoding. The indicates results using the original dataset prompts, which for the PIQA, ARC, and CSQA benchmark only contain the question. All other results use modified prompts, where the choices are provided with the question. The CSQA dataset has exactly 5 choices for each question, but we run the adversarial search for only 24 permutations. All scores are accuracies (%).
Model
Training
PIQA
ARC
CSQA
SIQA
Std.
Adv.
Std.
Adv.
Std.
Adv.
Std.
Adv.
Causal Mask+PE*
Pretrained
76.77
37.80
51.76
37.26
Causal Mask+PE"
Finetuned
79.82
45.39
68.80
75.95
Causal Mask+PE
Pretrained
57.45
30.96
36.03
7.68
34.92
16.46
39.29
12.74
Causal Mask+PEUM
Pretrained
68.31
50.49
43.18
14.93
45.21
27.35
46.72
16.84
Causal Mask+PE
Finetuned
84.11
76.77
55.20
23.72
78.31
69.62
74.80
63.00
Causal Mask+PEUlte
Finetuned
83.98
77.31
56.32
26.88
77.89
68.47
74.33
63.97
Prefix Mask+PE
Finetuned
82.78
76.50
57.62
27.47
78.98
71.01
74.36
65.66
Prefix Mask+PgUle
Finetuned
84.93
79.11
61.26
34.47
79.28
70.19
75.18
67.45
Prefix Mask+SetPE
Finetuned
81.23
81.23
51.28
51.28
77.31
77.31
71.24
71.24
Prefix Mask+SedPE
Finetuned
81.88
81,88
56.48
56.48
77.97
77.97
73.54
73.54
SetMask4SetPP
Finetuned
84,33
84.33
57.76
57.76
79.93
79.93
75.38
75.38
SetMask+SelPE
Finetuned
85,80
85.80
65.02
65.02
80.18
80.18
76.15
76.15
"Results with original prompts
Ulto Ultr Additional pretraining
fOnly first 24 (of 120) permutations tested
29

---

E.2
Majority vote
Additional results with majority vote are presented in Table
Table 15: Majority Vote results with Gemma 2B the base model. The indicates results using the original dataset prompts, which for the PIQA, ARC, and CSQA benchmark only contain the question. All other results modified prompts, where the choices provided with the question. Thee CSQA dataset has exactly choices for each question, but we run the adversarial search for only 24 permutations. All scores accuracies (%).
Model
Training
Eval. Mode
PIQA Std. Adv.
ARC
CSQA Std.
SIQA Std. Adv.
Std.
Adv.
Adv.
Causal Mask+PE* Causal Mask+PE*
Pretrained
Single run
76.77 79.82
37.80
51.76
37.26
Finetuned
Single run
45.39
68.80
75.95
Causal Mask+Pp Ult
Pretrained
Single run
68.31
50.49
43.18
14.93
45.21
27.35
46.72
16.84
Causal Mask+PPumm
Pretrained
Majority Vote
68.12
68.12
46.16
46.16
45.95
45.95
48.62 48.62
Causal Mask+PE Uitaa
Finetuned
Single run
83.98 77.31
56.32
26.88
68.47 77.89
74.33 63.97
Causal Mask+PP
Finetuned
Majority Vote
83.57 83.57
59.56
59.56
78.46 78.46
75.23 75.23
Causal Mask+NoPE
Finetuned
Single run
75.03 64.53
37.01
14.68
57.41 68.87
65.57 51.79
Causal Mask+NoPE
Finetuned
Majority Vote
75.35 75.35
38.82
38.82
69.45 69.45
66.63 66.63
Prefix Mask+PF
Finetuned
Single run
84.93 79.11
61.26
34.47
70.19 79.28
75.18
67.45
Prefix Mask+PFUllt
Finetuned
Majority Vote
85.36
85.36
64.08
64.08
79.77
79.77
75.84 75.84
SetMask-SetPP.
Finetuned
Single run
85.80 85.80
65.02
65.02
80.18 80.18 76.15 76.15
Results with original prompts
Ultra Additional pretraining
'Only first 24 (of 120) permutations tested
30

---

E.3 Different base LLMss
Additional results with different base LLMs are presented in Table 16 Table T7 provides the flip probabilities for the Causal Mask+PE and SetMask+SettE" models. The flip probability is measure how inconsistent the model predictions are between different permutations. Please, see Section E.4 for more details.
Table 16: Performance with different base LLMs. All results (except Pretrained") are (pre-)finettuee) on ultrafeedback dataset. The CSQA dataset has exactly choices for each question, but we run the adversarial search for only 24 permutations. All scores accuracies (%).
LLM
Model
Training
PIQA
ARC
CSQA Rand. Adv.
SIQA
Rand. Adv.
Rand. Adv.
Rand, Adv.
Causal Mask+PE'
Pretrained
76.77
37.80
51.76
37.26
Causal Mask+PP PE P
Pretrained
68.31 50.49
43.18 14.93
45.21 27.35
46.72
16.84
Gemma
Causal Mask+PE
Finetuned
83,98 77.31
56.32
26.88
77.89
68.47
74.33
63.97
+ Majority Vote
Finetuned
84.17 84.17
60.15
60.15
78.71
78.71
75.38
75.38
No Options
Finetuned
79.87 79.87
43.26
43.26
69.37
69.37
56.55
56.55
SetMask+SSePPE
Finetuned 85.80 85.80
65.02
65.02
80.18
80.18
76.15
76.15
Causal Mask+PE"
Pretrained 80.41
43.77
62.16
65.71
Causal Mask+PPF
Pretrained 86.21 78.67
79.68
56.23
69.96
47.17
70.34
49.69
Gemma 7B
Causal Mask+Pp
Finetuned 92.82 89.45
83.52
64.33
85.45
79.12
80.93
74.10
Majority Vote
Finetuned 92.66 92.66
85.58 85.58
85.75
85.75
81.10
81.10
+ No Options
Finetuned 83.51 83.51
52.47
52.47
73.79 73.79
59.47
59.47
SetMask-SSePPE:
Finetuned 92.98 92.98
83.45
83.45
84.93
84.93
81.12
81.12
Causal Mask+PE*
Pretrained 74.32
35.41
55.77
51.59
Causal Mask+PE Ultra
Pretrained 63.03 40.48
40.12
9.13
45.29
21.21
49.27
21.39
Llama 3.2 IB
Causal Mask+PEUIm
Finetuned 79.57 71.33
53.61
21.93
74.50
64.21
71.84
62.79
Majority Vote
Finetuned
79.49 79.49
57.17
57.17
75.51
75.51
71.85
71.85
No Options
Finetuned 77.42 77.42
39.33
39.33
65.03
65.03
53.63
53.63
SetMask+SetPE Ultra
Finetuned 81.66 81.66
59.30
59.30
76.66
76.66
72.47
72.47
Causal Mask+PE"
Pretrained 76.33
43.94
61.92
65.97
Causal Mask+PpUllne
Pretrained 76.41 64.09
68.83
39.93
66.85
44.31
66.56
47.80
Ilama 3.2 3B
Causal Mask+Pr
Finetuned
86.92 81.72
74.16
53.07
81.32
74.94
77.54
70.42
Majority Vote
Finetuned
86.83 86.83
76.37 76.37
81.57
81.57
77.99
77.99
No Options
Finetuned
79.54 79.54
45.31
45.31
71.34
71.34
56.55
56.55
SetMask+SedPE
Finetuned 88.41 88.41
75.85
75.85
83.29
83.29
80.30
80.30
Causal Mask+PE
Pretrained
80.09
53.41
66.50
69.34
Causal Mask+Pgllle Ultaa
Pretrained 83.30 72.36
78.75
56.66
72.67
53.81
70.91
54.96
Llama 3.1 8B
Causal Mask+PE Ulra
Finetuned
90.81 86.29
83.04
64.51
83.96
77.89
80.77
73.90
Majority Vote Finetuned 90.75 90.75
85.32
85.32
84.11
84.11
81.12
81.12
No Options
Finetuned 82.43 82.43
46.33
46.33
72.97
72.97
58.60
58.60
Set
Finetuned 91.62 91.62
84.13 84.13
85.34 85.34
81.47 81.47
"Results with original prompts
Ulra Additional pretraining
'Only first 24 (of 120) permutations tested
31

---

Table 17: We use flip probabilities to measure how consistent model predictions are across permuta- tions. They measure how often a model prediction deviates from its majority vote for sample. All models are (pre-)finetuned on the ultrafeedback dataset. "The CSQA dataset has exactly 5 choices for each question, but we calculate the majority vote and the flip probabilities using only 24 permutations. All probabilities are given as percentages (%).
LLM
Model
Training
PIQA
ARC
CSQA
SIQA
Gemma 2B
Causal Mask+ Mask+PEUUla Pit
Finetuned
6.66
21.45
5.40
6.35
SetMask+SetPE
Finetuned
0.00
0.00
0.00
0.00
Gemma 7B
Causal Pa:sk+PPUUL
Finetuned
3.37
7.92
2.94
4.32
SetMask++St1PF
Finetuned
0.00
0.00
0.00
0.00
LJama 3.2 1B
Causal Mask+PEUUlto
Finetuned
8.24
23.37
5.72
5.70
SetMask+SetPEE
Finetuned
0.00
0.00
0.00
0.00
Llama 3.2 3B
Causal Mask+Pp:ume
Finetuned
5.20
11.76
3.32
4.39
SetMask+
Finetuned
0.00
0.00
0.00
0.00
LJama 3.1 8B
Causal Mask+PEUlloo
Finetuned
4.52
7.87
3.23
4.63
SetMask+SSttEEUNn
Finetuned
0.00
0.00
0.00
0.00
E.4 Flip Probabilities
The flip probability the fraction of times that the predictions differ from the majority vote. Specifically, run all possible permutations of the input through the LLM and pick the option with the most "votes" as the majority vote. then count how many of the votes disagree with the majority vote and take the average over all samples. One can see this measure of how inconsistent the model predictions are across permutations. Unlike the adversarial setting, where single permutation of ls! can lead to incorrect outcome, the flip probability is less dependent on the value of k.
Table 17 shows the flip probabilities for the finetuned base models and their Set-LLM counterparts from the main table of results, Tables 3 and 16 The flip probability for Set-LIMM is guaranteed to be 0%, and this is confirmed by the empirical results. In contrast, the flip probabilities for the finetuned original models are between 3 and 23% highlighting that order bias impacts final performance. The flip probabilities are quite consistent across the PIQA, CSQA, and SIQA, although the number of permutations varies between 2 and 24, but the flip probabilities ARC are significantly higher. Generally, smaller models have higher flip probabilities.
We thank the anonymous reviewer for their suggestion to include flip probabilities as an additional metric.
32

---

E,5 Out-of-distribution performance
Comparison of the out-of-distribution performance finetuned Set-LLM and baseline models. The base model is Gemma 2B.
Table 18: Results on out-of-distributionn datasets using Gemma 2B as the base model. In-distribution
results are grayed out. All scores are accuracies (%).
Model
Finctune Dataset
PIQA
ARC
CSQA
SIQA Adv.
Std.
Adv.
Std.
Adv.
Std.
Adv.
Std.
Pretrained
57.45 30.96
36.03
7.68
34.92
16.46
39.29
12.74
Pretrained Ullra
68.31 50.49
43.18
14.93
45.21
27.35
46.72
16.84
Causal Mask+PE
83,98 77.31
54.45
26.79
41.52 61.51
59.68
56.79
35.11
Majority Vote
PIQA
83.57 83.57
56.74
56.74
61.51
58.34
58.34
SetMask+SedPE
85.80 85.80
58.02
58.02
63.47
63.47
56,86
56.86
Causal Mask+PEUtm
67.27 47.55
56.32
26.88
57.61
35.71
54,91
31.73
Maaooity Vote
ARC
66.92 66.92
59.56
59.56
59.38
59.38
56.86
56.86
SetMask+SetPEUumn
68.61 68.61
65.02
65.02
63.39
63.39
60.64
60.64
Causal Mask+PE
71.84 56.64
51.63
26.45
77.89
68.47
55.72
43.1 14
Majority Vote
CSQA
72.03 72.03
53.58
53.58
78.46
78.46
56.81
56.81
SetMask+SePPP.
71.49 71.49
55.38
55.38
80.18
80.18
58.96
58.96
Causal Mask+PE
71.55
54.68
53.52
27.30
64.76
45.86
74.33 63.97
Majority Vote
SIQA
71.33
71.33
55.55
55.55
65.85 65,85
75.23 75.23
SetMask+SedPE
74.16
74.16
56.83 56.83
67.73 67.73
76.15 76.15
Ulba Additional pretraining
OOlly first 24 (of 120) permutations tested
E.6 Impact of Set-LLM training to performance on OOD non-set tasks
In addition to testing the dutto--distribution performance on other set-input tasks, we also measure the influence of the Set-LLM finetuning non-multiple choice questions. Ideally, Set-LLM could still be used for non-set tasks without an impact on performance.
We chose two standard LLM benchmarks to test the effect of set-LLM finetuning on non-multiple- choice questions. For question-answering we use 1000 test questions from SQuAD v2 (35), and for machine translation, we use the English-German subset from Flores-200 6 record accuracy, and for Flores-200, we record COMET scores.
For SQuAD v2 we
As finetuned models, we picked models finetuned on PIQA (arbitrarily) and multi-document QA (27]. We test the models O-shot, 1-shot, and finetuned settings. We compare the original models with the Set-LLM models. PIQA models were finetuned on multiplechoice questions and therefore trained to copy a choice from the input to the output. This creates a strong bias that is likely to have a significant impact on OOD performance. Multi-document QA models were finetuned to answer question based on a of (retrieved) documents.
The results are presented in Tables 19 and 20 The PIQA models were finetuned to copy a choice from the input the output. We find that these models struggle with SQuAD in the O-shot and 1-shot (to a lesser extent) settings. The multi-document QA models perform better. Gemma 2B Set-L.LM, in particular, performs very poorly. For Flores-200, the differences are less pronounced.
Overall, we find that Gemma 2B SetMask+SelPElllin underperforms Gemma 2B Causal Mask+PEUlle in some settings, but not when finetuned. However, the Llama-based models generally do not have sigmificant gap between original and Set-LLM performance. This true for both datasets and across almost all evaluation settings. The exception is Llama 3.2 3B finetuned on PIQA in the O-shot and 1-shot settings.
We conclude that Set-LLM does not have negative impact on finetuned downstream performance. However, is more susceptible to overfitting to multiple-choice question answering when trained on such a task.
33

---

Table 19: Performance of models finetuned on multipee-choice dataset PIQA applied to standard non-set language tasks. We report accuracy scores for SQuAD v2 and COMET scores for Flores-200 EN-DE in O-shot, 1-shot, and finetuned settings.
LLM
Model
SQuAD v2
Flores-200 EN DDE
O-shot" *
1-shot"
finetuned
O-shot
I-shot
finetuned
Gemma 2B
Causal M
0.57
0.62
0.89
0.76
0.73
0.71
SetMask+SetPE
0.25
0.46
0.91
0.66
0.66
0.68
Llama
Causal Mask+PB P
0.64
0.83
0.71
0.68
0.70
SetM
0.58
0.63
0.90
0.69
0.65
0.70
Llama 3.2 3B
Causal Mask+PBUlta
0.79
0.76
0.91
0.80
0.77
0.79
SetMask+SetPE.um
0.70
0.69
0.92
0.72
0.69
0.77
*Unanswerable questions were removed O-shot and 1-shot settings (SQuAD v2)
Table 20: Performance of models finetuned on Multi-documenn question answering 27 applied to
standard non-set language tasks. We report accuracy scores for SQuAD v2 and COMET scores for Flores-200 EN-DE in 0-shot, I-shot, and finetuned settings.
LLM
Model
SQuAD
Flores-200 EN
-NE
O-shot"
I-shot"
finetuned
O-shot
1-shot
finetuned
Gemma
Causal Mask+PEUhh
0.69
0.70
0.88
0.66
0.67
0.71
SetMask+SedPE
0.51
0.51
0.87
0.62
0.64
0.68
Llama 3.2 IB
Causal Mask+PEUlm Ulira
0.75
0.74
0.84
0.69
0.65
0.69
SetMask+SSttPE
0.75
0.72
0.90
0.71
0.68
0.69
Llama 3.2 3B
Causal Pulk+
0.85
0,82
0.91
0.78
0.74
0.79
SetMask+SerPP 0.83
0.81
0.93
0.75
0.73
0.78
"Unanswerable questions were removed for 0-shot and 1-shot settings (SQuAD v2)
E.7 Runtimes & memory usage
Table 21 shows the runtimes and memory usage for Causal Mask+PE and SetMask+SetPE models with Gemma base models. All models were trained and evaluated on Nvidia H200 GPUs on an internal cluster.
The total GPU time for the paper is estimated to be around 1200 hours on Nvidia H200 GPUs.
Table 21: Number of model runs per input, pre-fineunning time on Ultrafeedback, and finetuning and evaluation times on ARC-Challenge. Runtimes are calculated with Gemma 2B as the base model. k is the number of (multiple) choices in the input question. Memory usage is for finetuning on ARC.
LLM
Model
No. Runs per Input
Pretraining on UltraFeedback (s)
Finetuning on
Memory
Evaluation on ARC (s)
ARC (s)
Usage (GB)
Causal Mask+PE
1
6083.50
4880.68
13.73
357.63
Gemma 2B
+ Majority Vote M!
8620.81
SetMask+SetPE
1
6002.45
4908.22
13.61
365.47
Causal Mask+PF 1
9983.51
7852.38
40.37
331.41
Gemma 7B
Majority Vote
k!
8448.37
SetMask+SetPE
1
10003.62
7821.80
40.02
329.77
34

---

E,8
Majority vote vs Set-LLM outputs
When calculating the majority vote, the vote count could be considered as measure of model confidence. In Figures 9 we compare vote count with accuracy and see that higher vote counts indeed exhibit higher accuracies on average, though the relationship not as clear-cut for Commonsense(A..
We explore whether Dermutation-invarian model makes similar predictions to the hase model, in particular for samples with high vote counts. Figures 6 to 9 (right) shows the agreement rate between the Gemma 2B Causal Mask+PE baseline and SetMask+SelPEUln We see similar trend to the accuracy, whereby the agreement is higher for high vote count samples. However, the overall agreement rate of (0.6817) on ARC-Challenge demonstrates that the models often make different mistakes, Indeed the agreement rate when Causal Mask+PE + Majority Vote is incorrect is only around 50% for three of the four datasets.
Accuracy
Agreementt
10
1.0
fepremeer
0.4
0.2
0.2
Agreementt (Corract)= 5.9 5.9334
Agreement (Incoract) = 0.5 2.5196
Overall Acturacy = 0.8335
Overall Agreement = 0.8645
0.0
0.0
Majortt Vatle
Maarity
+ Majority Vote vote count versus accuracy (left). Causal Mask+ PD Pl + Majority Vote vote count versus agreement rate with Set-LLM (right) on PIQA.
Figure 6: Causal Mask+
Accuracy
Agreement
20
1.0
0.4
0.2
0.2
Agreament (Corrat))= 0.8511
Agreament (Incoracti = D. 0.4261
Overail Acturacy C 6015
Cverail Agreement - 0.6817
0.0
0.0
24
Majority Valte Crean
Maaniyy Vote Daun:
Figure 7: Causal Pask+ Majjority Vote vote count versus accuracy (left). Causal Mask+ MlakkPPUllng P + Majority Vote vote count versus agreement rate with Set-LLM (right) on ARC.
35

---

Arturayy (SDA
Agrnement Sptil Count -Snn--CSQA
0.
0.0
0.2
Agreemeett (Correc))= 0.9229
Agreement (Incorrect) - 0.5249
Overall Accuracy 0.7862
Cverall Agreement 0.8378
0.0
0.0
Figure 8: Causal Mask+PE + Majority Vote vote count accuracy (left). Causal
Mask+PE Ult +Majority Vote vote count versus agreement rate with Set-LLM (right) on CSQA.
Accuracy SIDA
Agreement with SIGA
0.
0.0
0.2
Agg eemeenl (Qurreu))= 0.9261 #greement (Incorrect) = ruse Overall Agreement = 0.8736
Overal Accuracy 0.7569
0.0
0.
Maganty Vitte Count
Wayorty
Figure 9: Causal Mask+PEllie Majjor Vote vote count accuracy (left). Causal Mask+ PUUni +Majority Vote vote count versus agreement rate with Set-LLM (right) on SIQA.
36

---

80
70
LLama 3.2 1B, Causal Mask+PE"M.
Accuracy
60
LLama Causal Mask+PEUka
LLama 3.2 1B, SetMask SctMask-SeuPE She
LLama 3.2 3B, Causal Pas
50
ILaama Causal Mask+PEUhn
LLama 3.2 3B, SetMask+SeePE
40
GPT-b.5-Turo 2H
30
Position of Document with the Answer
Figure 10: Varying the position of the relevant document (containing the answer) within the pretrained language models' (LLama... Pretrain and GPT-3.5-Turbo inputs results in U-shaped performance curves. However, the Set-LLM adapted models produce completely flat performance curves with higher accuracy. Finetuning the base model results a mostly flat and similarly high performance curve. -GPTo3.-TTurb was not finetuned the dataset.
E.9 Additional multi-document question answering results
The results in Figure I0 reveal a less pronounced U-shaped curve for the pretrained base model (Llama 3.2 3B) than those observed in previous work (27). The finetuned base model again produces mostly flat curve with no clear evidence of position bias. The Set-LLM curve is completely flat as expected, and scores slightly higher on average than the finetuned base model.
E.10 LLM-as-a-judge
We initial LLM-as- LLM-as-a-judge experiments showcase the potential of Set-LLM as permutation- invariant evaluator. We train and test our models on the Search Arena dataset [31]. We restrict the search arena dataset questions with single turn (not multiturn), with winner (no ties), and no longer than 5000 characters. We use a 2/3-1/3 1/3 train-test split.
We use the smaller Gemma and LLama models from the rest of the paper. However, it should be noted that LLM evaluators/judges are typically large, powerful models whose output can be used for both evaluation and finetuning. We report accuracy scores with respect to human judgments. The results are shown in Table
We see that in all cases, the Set-LLM model closely matches the baseline model in vandom-orde performance, and significantly outperforms it in the adversarial setting. The performance of the baseline models even drops below random guessing in the adversarial setting, Although the results are promising, a more thorough study should be conducted with realistic LLM evaluator model sizes.
Table 22: L.LM-as a-adde application with different base LLMs. All models were pretrained on UltraFeed-back [7] and then finetuned and tested on Search Arena [31. All metries are accuracies to two significant figures.
LLM
Model
Rand,
Adv.
Rand. Baseline
0.50
0.50
Gemma 2B
Causal Mask+PELton
0.58
0.40
SetMask+ SeiP
0.57
0.57
Llama 3.2 1B
Causal Mask+PpUIm
0.57
0.38
SetMask+SetPEUUn
0.60
Llama 3.2. 3B
Causal Mask+Pplit
0.62
0.47
SerMask-SePPr.thn
0.62
0.62
37


Reviews:
[{"review_id": "uJqoKOWbCA", "reviewer": "Reviewer_Zm9C", "summary": "This paper introduces Set-LLM, a permutation-invariant adaptation of decoder-only large language models (LLMs) designed to eliminate order sensitivity when processing mixed set-text inputs, such as multiple-choice options or sets of candidate responses. The authors provide theoretical proofs and comprehensive empirical results demonstrating that Set-LLM improves robustness to input permutations while maintaining or surpassing the performance of baseline models.", "questions": "- Table 1 and Table 3, which report performance across models, should be merged to facilitate clearer, side-by-side comparisons of model sizes and methods.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper tackles a clear and impactful vulnerability in LLMs—order sensitivity—by introducing a novel architectural solution that guarantees permutation invariance.\n-The authors provide formal proofs of permutation invariance, offering a solid theoretical foundation for the proposed architecture.\n-Experiments across five models and four multiple-choice datasets validate the effectiveness and generalization of Set-LLM.\n\n**Weaknesses:**\n\n- Some technical sections (e.g., the formalization of set-token mappings in Lines 104–107) are notation-heavy and complex and may benefit from clearer exposition or illustrative examples.\n- While the multiple-choice QA setting is well-justified, experiments focus solely on this task; broader generalization to other set-input scenarios (e.g., retrieval-augmented generation, ranking) is left untested.\n- Including baselines that do not provide input choices (i.e., original dataset prompt) seems tangential to motivation and are not sufficiently analyzed in the experimental section.", "was_revised": true, "final_justification": "After the rebuttal, my main concerns about the task's generalization ability were resolved. I lean toward acceptance of this paper.", "rebuttal": "We sincerely thank Reviewer Zm9C for the positive feedback and valuable suggestions to improve the paper. \n\n\n> W1: Some technical sections (e.g., the formalization of set-token mappings in Lines 104–107) are notation-heavy and complex and may benefit from clearer exposition or illustrative examples.\n\n\nThank you for raising this concern. We have tried to include figures and illustrative examples to help the reader, but we will certainly go through the paper again and see if we can improve clarity. Indeed, the set-token mappings sound much more complicated than they are, and we included Figure 3 for this reason. We will take a close look at whether we can simplify this formalization for the final version. We hope the intuition however is clear.\n\n\n> W2: While the multiple-choice QA setting is well-justified, experiments focus solely on this task; broader generalization to other set-input scenarios (e.g., retrieval-augmented generation, ranking) is left untested.\n\n\nWe performed additional experiments on retrieval-augmented generation and multi-document summarization. We describe these and discuss the results below.\n\n## Multi Document Summarization\n\nWe conduct our evaluation on the MultiNews [1] dataset. We filter out inputs of length greater than 20000 characters to satisfy our memory constraints. We use the standard train-validation-test split. We compare the original model architecture with Set-LLM with single runs (no adversarial setting). Our initial results do not include hyperparameter tuning, we use the best-performing hyperparameters from the multiple-choice experiments. The experiments will be expanded for the final paper.\n\nWe report standard metrics, this includes Rouge F1 scores and compression - the length of the original text divided by the length of the summary. We also include Rouge F1 scores between the target summaries and the model summaries (R1*, R2*, R3*).\n\n|Model             |Variant        |Compression |R1      |R2      |RL      |R1*      |R2*     |RL*    |\n|-------------------|----------------|-------------------|---------|---------|---------|---------|---------|---------|\n|Target             |                    |5.18               |0.30    |0.14    |0.16    | -        | -         | -        |\n|Gemma 2B     |Original       |13.21             |0.19    |0.14    |0.15    |0.32    |0.10    |0.19   |\n|                       |Set-LLM      |6.39               |0.30    |0.23    |0.23    |0.48    |0.19    |0.26   |\n|Gemma 7B     |Original       |8.11               |0.27    |0.21    |0.22    |0.42    |0.15    |0.23   |\n|                       |Set-LLM      |6.64               |0.30    |0.23    |0.23    |0.50    |0.22    |0.28   |\n|Llama 3.2 3B  |Original       |8.86               |0.25    |0.17    |0.18    |0.42    |0.14    |0.22   |\n|                       |Set-LLM      |6.31               |0.30    |0.20    |0.19    |0.50    |0.20    |0.26   |\n\nSet-LLM outperforms the finetuned baseline with all three base models in all metrics. However, the margin is significantly narrower with the larger Gemma 7B model. Note that the compression rate is always lower for Set-LLM, meaning the summaries it produces are longer.\n\n[1] Fabbri, Alexander R., et al. \"Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.\" arXiv preprint arXiv:1906.01749 (2019).\n\n\n## Multi-Document Question Answering (Retrieval-Augmented Generation)\n\nMotivated by [2], we evaluate Set-LLM on retrieval-augmented question answering. The authors introduce a task mimicking the retrieval-augmented generation setup. They ask questions with 20 supporting documents, where only one document contains the information required for the answer. They show that commercial LLMs (GPT3.5 Turbo) produce a U-shaped performance graph given the location of the answer. If the document with the answer is listed first or last, then the LLM answers the question correctly with a much higher probability (e.g. 75% in position 1/20 vs 55% in position 10/20). In contrast, Set-LLM is guaranteed to have a flat performance curve, since reordering the documents does not affect the outcome. \n\nWe split the dataset into 60% training, 10% validation, and 30% test data. We replicate the training data 5 times and shuffle the documents randomly for each input. This is done to avoid the model overfitting to a specific formulation of each training question. We have not conducted hyperparameter tuning yet, the hyperparameters were taken from the multiple-choice experiments.\n\nWe have 5 different test sets with the answer document placed in 5 different locations (first, 5th, 10th, 15th, last). We provide the results for two LLama-based models below. All scores denote the percentage accuracy.\n\n|Model             |Variant        |1         | 5        | 10     | 15      | 20      |\n|-------------------|----------------|---------|---------|---------|---------|---------|\n|Llama 3.2 1B  |Original      |34.76  |26.85  |25.22  |28.23  |37.39  |\n|                       |Set-LLM     |76.54  |76.54  |76.54  |76.54  |76.54  |\n|Llama 3.2 3B  |Original      |54.83  |44.29  |44.92  |44.04  |52.07  |\n|                       |Set-LLM     |81.68  |81.68  |81.68  |81.68  |81.68  |\n\n[2] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173.\n\n\n> W3: Including baselines that do not provide input choices (i.e., original dataset prompt) seems tangential to motivation and are not sufficiently analyzed in the experimental section.\n\n\nThis is a great suggestion. We have run experiments with a pretrained model and with a model (pre-)finetuned on the ultrafeedback dataset and then finetuned on the multiple choice datasets separately. We finetuned these models again, because the dataset now doesn't include the choices in the question. We report results below for Gemma 2B, but we will include results for all models in the final paper. There is no adversarial setting for this baseline. We include Set-LLM scores for comparison. We see that our model clearly outperforms this additional baseline.\n\n\n|Variant                           |PIQA   |ARC  |CSQA |SIQA  |\n|--------------------------------|---------|---------|---------|---------|\n|Causal Mask+PE           |76.77  |37.80  |51.76  |37.26 |\n|Causal Mask+PE Ultra  |79.87  |43.26  |69.37  |56.55 |\n|Set-LLM + Ultra             |85.80  |65.02  |80.18  |76.15 |"}, {"review_id": "186eGNj68u", "reviewer": "Reviewer_ryD4", "summary": "This paper proposed set-llm, a technique that can help finetune base llm to be permutation invariant. The key method is here to introduce a new attention masking and a new set position embedding.\nExperiments are mainly conducted on multiple choice evaluations with random and adversarial permutations of the choices to test the robustness of the proposed method to permutated choices.\nThe results suggest that the proposed method indeed is robust to different permutations.", "questions": "See weakness above.\nI feel that the weakness overweights the strength for this work.", "limitations": "The paper has discussed the limitations", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n\nThe overall approach towards the solution for permutation invariant models is straightforward and effective.\nThe experiments agree with the theory behind this work.\nThe paper is well written and includes enough ablation studies to test the performance of the proposed model.\n\n\n\nWeakness\n\nThe scope for this work seems to be rather limited, only targeting the multiple choice QA questions. Consider the method requires finetuning to the base model, I would question if this is necessary to finetune a model just to make it work for multi-choice questions.\n\nAdditionally, the paper has shown that the proposed finetuning is able to improve on multiple choice question, I wonder what's the influence of the finetuning process to non-multiple choice questions? Such as code generation tasks measured by HumanEval or other benchmarks.", "was_revised": true, "final_justification": "I would raise my score to 4 considering the additional experiments demonstrated. I still have reseverations on the applicability to wider applications, but I think this paper at least should be interesting when considering for specific applications like LLM-as-a-judge.", "rebuttal": "We sincerely thank Reviewer ryD4 for their feedback and for helping us to improve our paper. \n\n\n> W1: The scope for this work seems to be rather limited. \n\n\nWe understand the concern. We argue in the paper that the multiple choice setting is a reasonable test case for LLM-as-a-Judge applications, and LLM-as-a-Judge has wide-ranging applications. We mention other possible applications in the paper and see our model as a possible blueprint for these applications. \nTo strengthen our paper and showcase the versatility of our approach we have conducted additional experiments on multi document summarization and retrieval augmented generation, both common LLM tasks. The results validate the general applicability and benefits of the proposed method. We have not been able to run all the baseline models on both of these applications, but we will make sure to complete these results for the final version of the paper.\n\n\nTo strengthen our paper and showcase the versatility of our approach we have conducted additional experiments on multi document summarization and retrieval augmented generation, both common LLM tasks. The results validate the general applicability and benefits of the proposed method. We have not been able to run all the baseline models on both of these applications, but we will make sure to complete these results for the final version of the paper. \n\n## Multi Document Summarization\n\nWe conduct our evaluation on the MultiNews [1] dataset. We filter out inputs of length greater than 20000 characters to satisfy our memory constraints. We use the standard train-validation-test split. We compare the original model architecture with Set-LLM with single runs (no adversarial setting). Our initial results do not include hyperparameter tuning, we use the best-performing hyperparameters from the multiple-choice experiments. The experiments will be expanded for the final paper.\n\nWe report standard metrics, this includes Rouge F1 scores and compression - the length of the original text divided by the length of the summary. We also include Rouge F1 scores between the target summaries and the model summaries (R1*, R2*, R3*).\n\n|Model             |Variant        |Compression |R1      |R2      |RL      |R1*      |R2*     |RL*    |\n|-------------------|----------------|-------------------|---------|---------|---------|---------|---------|---------|\n|Target             |                    |5.18               |0.30    |0.14    |0.16    | -        | -         | -        |\n|Gemma 2B     |Original       |13.21             |0.19    |0.14    |0.15    |0.32    |0.10    |0.19   |\n|                       |Set-LLM      |6.39               |0.30    |0.23    |0.23    |0.48    |0.19    |0.26   |\n|Gemma 7B     |Original       |8.11               |0.27    |0.21    |0.22    |0.42    |0.15    |0.23   |\n|                       |Set-LLM      |6.64               |0.30    |0.23    |0.23    |0.50    |0.22    |0.28   |\n|Llama 3.2 3B  |Original       |8.86               |0.25    |0.17    |0.18    |0.42    |0.14    |0.22   |\n|                       |Set-LLM      |6.31               |0.30    |0.20    |0.19    |0.50    |0.20    |0.26   |\n\nSet-LLM outperforms the finetuned baseline with all three base models in all metrics. However, the margin is significantly narrower with the larger Gemma 7B model. Note that the compression rate is always lower for Set-LLM, meaning the summaries it produces are longer.\n\n[1] Fabbri, Alexander R., et al. \"Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.\" arXiv preprint arXiv:1906.01749 (2019).\n\n\n## Multi-Document Question Answering (Retrieval-Augmented Generation)\n\nMotivated by [2], we evaluate Set-LLM on retrieval-augmented question answering. The authors introduce a task mimicking the retrieval-augmented generation setup. They ask questions with 20 supporting documents, where only one document contains the information required for the answer. They show that commercial LLMs (GPT3.5 Turbo) produce a U-shaped performance graph given the location of the answer. If the document with the answer is listed first or last, then the LLM answers the question correctly with a much higher probability (e.g. 75% in position 1/20 vs 55% in position 10/20). In contrast, Set-LLM is guaranteed to have a flat performance curve, since reordering the documents does not affect the outcome. \n\nWe split the dataset into 60% training, 10% validation, and 30% test data. We replicate the training data 5 times and shuffle the documents randomly for each input. This is done to avoid the model overfitting to a specific formulation of each training question. We have not conducted hyperparameter tuning yet, the hyperparameters were taken from the multiple-choice experiments.\n\nWe have 5 different test sets with the answer document placed in 5 different locations (first, 5th, 10th, 15th, last). We provide the results for two LLama-based models below. All scores denote the percentage accuracy.\n\n|Model             |Variant        |1         | 5        | 10     | 15      | 20      |\n|-------------------|----------------|---------|---------|---------|---------|---------|\n|Llama 3.2 1B  |Original      |34.76  |26.85  |25.22  |28.23  |37.39  |\n|                       |Set-LLM     |76.54  |76.54  |76.54  |76.54  |76.54  |\n|Llama 3.2 3B  |Original      |54.83  |44.29  |44.92  |44.04  |52.07  |\n|                       |Set-LLM     |81.68  |81.68  |81.68  |81.68  |81.68  |\n\n[2] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173.\n\n\n\n\n> W2: I would question if this is necessary to finetune a model just to make it work for multi-choice questions.\n\n\nTo add to the response to W1, the finetuning cost is fairly small (between 1 and 4 GPU hours) for each dataset and model we consider. This is negligible in comparison to the millions of GPU hours used to pretrain these models (e.g. 1.46M GPU hours for llama 3.1 8B). Moreover, the main purpose of finetuning is to allow the model to adapt to its new, altered architecture. We envisage that a typical scenario would involve finetuning Set-LLM on a general-purpose set-dataset and then using it for relevant downstream applications."}, {"review_id": "E2OiqudQ8b", "reviewer": "Reviewer_nn3i", "summary": "In this work, the authors present Set-LLM, a novel architectural modification to pretrained large language models designed to mitigate their sensitivity to input order. Specifically, they propose changes to the positional encoding and attention masking mechanisms to enable the model to handle mixed set-text inputs—such as text sequences containing permutable components like option lists—while ensuring permutation invariance. They evaluate their proposed dataset on various MCQ benchmarks and with different models, ranging from 1B to 8B.", "questions": "- Concerning the Set-Mask, I understand that the permutable options should not attend to one another, but why change the causal nature of the mask as well, especially since that can cause severe changes to the model behaviour ? \n- There seems to be a missing notation in the statement of Theorem 1, what is \\tilde{X}(t+1) ?\n- In Table 2, it is surprising that the PrefixMask + No positional embeddings configuration still yields meaningful outputs. Without positional information, the model essentially becomes a bag-of-words processor. Could the authors explain how such a setup maintains coherent output?\n- Concerning the adversarial order: do the authors generate multiple completions per input order, or just one? If only a single output is used per permutation, then it becomes difficult to disentangle the effects of input order from the inherent randomness of LLM outputs. Even with a fixed order, generating n samples will eventually yield some incorrect outputs due to sampling variance. In this context, changing the order could function similarly to changing the random seed, complicating the interpretation of the adversarial metric.", "limitations": "See above", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I found this paper interesting and well-written. The exposition is clear, making the work easy to read and follow.\n\nRelevance: \nThe problem of order sensitivity in large language models is timely and important. The proposed approach addresses the issue in a meaningful way. The experimental section is convincing, leveraging multiple models and benchmarks to support the claims. However, I do have a few remarks and questions (detailed below).\n\nLimitations: \n- A potential limitation is that the proposed modifications to the attention mask and positional embeddings require an additional training step. That said, the computational overhead seems manageable (reported as up to 4 hours on a single H200 GPU for 8B models).\n- Switching from a causal mask to a prefix mask could significantly affect model behavior. While this may not be apparent in tasks like multiple-choice questions (MCQs), where outputs are short and structured. It would strengthen the paper to verify that general model capabilities do not degrade on standard benchmarks.\n- On a side note, I am not entirely convinced the theorems qualify as such, especially the second one, which seems rather straightforward.", "was_revised": true, "final_justification": "This paper proposes Set-LLM, an architectural modification to improve permutation invariance in LLMs for mixed set-text inputs. The problem is timely and well-motivated, and the approach is clearly presented with solid empirical results across multiple models and benchmarks. While the required fine-tuning introduces additional training cost, the overhead seems manageable, and the authors’ additional experiments confirm minimal degradation on general benchmarks. Some methodological choices, such as removing causal masking, could be further explored, but overall, the work offers a meaningful and well-supported contribution.", "rebuttal": "We sincerely thank Reviewer nn3i for the positive feedback and thoughtful suggestions. \n\n> W1: Requires finetuning. That said, the computational overhead seems manageable.\n\nWe agree with the reviewer, that the proposed fine-tuning method results in an additional training step, however the amount of fine-tuning (4h on a single GPU) is negligible in comparison to the overall pre-training time (e.g. 1.46M GPU hours for llama 3.1 8B) and the resulting model can be used in a range of applications.\n\n> W2: It would strengthen the paper to verify that general model capabilities do not degrade on standard benchmarks.\n\nWe endeavour to provide an evaluation of the models after fine tuning on standard evaluation benchmarks they were built for during the author reviewer discussion period. We have machine translation and math in mind, but would be happy to take suggestions. \nNote, that in response to other reviewers, we have evaluated Set-LLM on additional tasks (multi-document summarization and retrieval-augmented generation).\n\n> Q1: Why change the causal nature of the mask?\n\nThis is an interesting suggestion. The overall causal structure of the mask has to be destroyed in order to guarantee invariance. However, the attention within an option could remain causal. We have not tried this, but it would be an interesting addition to our ablation studies. We do not expect significant performance gains since a fully-connect prefix mask is shown to be at least as effective as a causal mask in [1].\n\n[1] Kopiczko, Dawid J., Tijmen Blankevoort, and Yuki M. Asano. \"Bitune: Bidirectional instruction-tuning.\" arXiv preprint arXiv:2405.14862 (2024).\n\n> Q2: There seems to be a missing notation in the statement of Theorem 1, what is \\tilde{X}(t+1) ?\n\nThank you for bringing this to our attention. \\tilde denotes variables after permuting the input. We will make sure to add this clarification.\n\n> Q3: In Table 2, it is surprising that the PrefixMask + No positional embeddings configuration still yields meaningful outputs.\n\nIndeed, this is very surprising, that a bag-of-words, or in this case better a bag-of-tokens model performs so well, especially the 74.54% on PIQA. This probably says more about the dataset than about the model. Note that the performance on the other datasets is weaker.\n\n> Q4: Concerning the adversarial order: do the authors generate multiple completions per input order, or just one?\n\nWe do not generate completions as answers, instead, the possible options are ranked by their log probabilities. Therefore the evaluation is deterministic once the model has been trained."}, {"review_id": "eAoeBw7uDR", "reviewer": "Reviewer_f4vF", "summary": "The predictions of LLMs generally depends on the order of the tokens passed as input. While this is generally desirable for sequence modeling, it is undesirable in cases where the LLMs is presented with a set of inputs whose order is not relevant such as LLM-as-a-judge or multiple choice question answering.\n\nThis paper presents architecture modifications that allows to fine-tune an LLM so that it can handle tokens whose are arbitrary. Experiments are conducted on multiple-choice formulation benchmarks which shows the proposed method can perform predictions equivariant to the order of set inputs.", "questions": "While the paper is very well written and the motivation is relevant, I believe the application is very restricted and the experiments are potentially biased toward emphasizing the problem the paper tackle.\n\n**Restricted application.**\n\nRegarding the application being restricted, you mention LLM-as-a-judge and multiple choice formulation (MCF). However, as you rightly point out, LLMs can be made equivariant by averaging the predictions under all possible permutations. While I agree that it leaves the application useful for MCF which has often 4 choices, the case for LLM-as-a-judge is limited as LLMs just needs to average two options. I think this strongly limits the impact for LLM-as-a-judge as the cost of 1) having a custom position encoding / attention mask 2) requiring a custom fine-tuning is likely to far exceed the the benefit of saving a 2x factor in the cost.\n\nRegarding MCF, I agree though that averaging all 24 permutations would incur substantial extra cost. However, this application is restricted too given that MCF are most often used for evaluations and not much from typical user queries. Furthermore, it is only relevant to benchmarks using MCF and not cloze formulations which further restricts the scope.\n\nIn addition to the limited set of applications, the method requires fine-tuning with custom mask and positional embeddings which would also limit the adoption.\n\n**Bias of experiments.**\nRegarding the bias of experiments, I believe the paper overemphasises the issue of the order in several ways. First, I believe the metric chosen is a worse case scenario since you pick the one permutation among 24 adversarially. I think this metric does not really show the probability that an inversion occurs (all models will get low adversarial scores when $k!$ becomes large). I would find the results more convincing if you report the fraction of times the predictions differ from the mode as it is probably closer from what a LLM user cares about.\n\nSecond, I think the problem you mention of having predictions swapped with the input occur a lot more for small models. This can be seen in Tab 3. for instance where the gap between random and adversarial seems to decay with model size (eg when going from 2B -> 7B and 1B -> 8B). I believe this should be properly analyzed and mentioned in the paper, ideally including a 70B model as the scope of the paper would be reduced if the problem occur just for small models.\n\nI would consider improving my score if they are other applications with a sufficiently large set size so that the method is relevant, if there is a clear evidence that the swapping happens enough and does not vanish to a large extent when using larger models.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and easy to understand\n* The experimental setup is clear and well described\n\nWeaknesses:\n* The paper targets a niche aspect of LLMs and may have little to no practical use\n* In addition to having a potentially limited impact, the solution proposed by the author appears very simple (modify the positional encoding and the attention mask) and requires fine-tuning the model\n* The experiments while being correct are biased towards the result the author want to show", "was_revised": true, "final_justification": "I raised my score significantly from 2 to 5. This is because the author addressed my main critics: the potential lack of scope/impact and the way results may be biased.\n\nRegarding the first one, they included experiments on relevant use-cases with much larger set cardinalities where 1) an equivariant approach would indeed be more beneficial (as one cannot average all the k! permutations) 2) significant improvements were shown over the baselines.\n\nRegarding the potential bias, I think the initial (adversarial) metrics chosen by the authors was not very illustrative of what an end-user will care about (eg what is the probability of having a different answer when permuting the output. They mention they will use a better metric and also now discuss clearly the fact that positional bias tends to diminish with larger model which is an important limitation.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper addresses an important problem with transformers: that in certain settings, such as multiple-choice-question (MCQ) answering, the LLM's choice should not depend on the order in which the answer options are presented.  In practice, it does depend on the order, which reduces the utility of MCQ as a benchmark. This paper addresses the problem with a suite of technical innovations in the low-level transformer architecture to make "set permutation invariance" a first-class capability of LLMs, and demonstrate that the techniques do in fact improve permutation invariance.

While the authors had some initial concerns with the paper, especially regarding limited applicability / scope of the method, most of the concerns were resolved with a robust author-reviewer discussion. The reviewers are now satisfied with the paper. Given its general potential interest to the field, and given the reviewers' positive ratings, I recommend acceptance.
Decision: Accept (poster)