Title: Order-Independence Without Fine Tuning
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLMs

Multiple Choice Questions

Transformers

Positional Encodings

Modified Attention Mask
Abstract: The development of generative language models that can create long and coherent textual outputs via autoregression has lead to a proliferation of uses and a corresponding sweep of analyses as researches work to determine the limitations of this new paradigm. Unlike humans, these '*Large Language Models*' (LLMs) are highly sensitive to small changes in their inputs, leading to unwanted inconsistency in their behavior. One problematic inconsistency when LLMs are used to answer multiple-choice questions or analyze multiple inputs is *order dependency*: the output of an LLM can (and often does) change significantly when sub-sequences are swapped, despite both orderings being semantically identical. In this paper we present , a technique that *guarantees* the output of an LLM will not have order dependence on a specified set of sub-sequences. We show that this method *provably* eliminates order dependency, and that it can be applied to *any* transformer-based LLM to enable text generation that is unaffected by re-orderings. Delving into the implications of our method, we show that, despite our inputs being out of distribution, the impact on expected accuracy is small, where the expectation is over the order of uniformly chosen shuffling of the candidate responses, and usually significantly less in practice. Thus, can be used as a '*dropped-in*' method on fully trained models. Finally, we discuss how our method's success suggests that other strong guarantees can be obtained on LLM performance via modifying the input representations.

Code is available at [github.com/reidmcy/set-based-prompting](https://github.com/reidmcy/set-based-prompting.).

Order-Independence Without Fine Tuning

Abstract

The development of generative language models that can create long and coherent textual outputs via autoregression has lead to a proliferation of uses and a corresponding sweep of analyses as researches work to determine the limitations of this new paradigm. Unlike humans, these ' Large Language Models ' (LLMs) are highly sensitive to small changes in their inputs, leading to unwanted inconsistency in their behavior. One problematic inconsistency when LLMs are used to answer multiple-choice questions or analyze multiple inputs is order dependency : the output of an LLM can (and often does) change significantly when sub-sequences are swapped, despite both orderings being semantically identical. In this paper we present Set-Based Prompting , a technique that guarantees the output of an LLM will not have order dependence on a specified set of sub-sequences. We show that this method provably eliminates order dependency, and that it can be applied to any transformer-based LLM to enable text generation that is unaffected by re-orderings. Delving into the implications of our method, we show that, despite our inputs being out of distribution, the impact on expected accuracy is small, where the expectation is over the order of uniformly chosen shuffling of the candidate responses, and usually significantly less in practice. Thus, Set-Based Prompting can be used as a ' dropped-in ' method on fully trained models. Finally, we discuss how our method's success suggests that other strong guarantees can be obtained on LLM performance via modifying the input representations.

Code is available at github.com/reidmcy/set-based-prompting .

1 Introduction

Recent advances in machine learning have led to a paradigm shift, as new training methods, which rely on massive scale datasets to create general purpose 'base models', outperform specialized single purpose models (Achiam et al., 2023; Radford et al., 2019). A particularly notable development is in natural language processing, where self-supervised learning has led to a cornucopia of transformer based models that can string together multiple next

a) Llama 2, 7B

The pitch of a sound is

mostly related to its









'intensity'

'frequency'

'amplitude'

'speed'









Answer:

frequency

2 GLYPH<8>

b) Llama 2, 7B

Reversed Order

The pitch of a sound is

mostly related to its









'speed'

'amplitude'

'frequency'

'intensity'









Answer:

amplitude

4

c) Llama 2, 7B

Set-Based Prompting

The pitch of a sound is

mostly related to its













'speed'

'amplitude'

'frequency'

'intensity'















Answer:

frequency

2 GLYPH<8>

token predictions, via autoregression, to generate coherent text that approximates a human response. These generated responses are not human and exhibit non-human behavioral quirks due to the limitations of the transformer based LLM architecture. Some of these, like the inability to reason about letters, are due to the choice of tokenization (the process of converting text into a sequence of numbers that can be used as an input) which breaks words into multiple character chunks (McCoy et al., 2023), while others like the Lost Middle Phenomenon are limitations of the training data (N. F. Liu et al., 2024). The solutions to these limitations are known (An et al., 2024), although often impractical (Sutton, 2019).

There are other limitations of LLMs that appear to be fundamental to the design of LLMs. One pernicious issue is the order dependency problem . This is the phenomenon where the output of an LLM can be significantly altered by changing the order of inputs, even when the re-ordering should have an identical response; see figure 1 for an example. This order dependency problem is well studied for multiple choice questions (Pezeshkpour et al., 2023; Zheng et al., 2024), but can happen on any task.

Having systems that make or aid in decision making that are sensitive to factors of the input that are orthogonal to the goals of the operators presents significant concerns. Consider an LLM that reads medical papers and summarizes them to a doctor (Van Veen et al., 2024). If shuffling the papers changes the summary can the doctor trust the model's responses? Additionally, order dependency presents algorithmic fairness concerns, such as in a case where an LLM being used to compare candidates has a slight bias for the first candidate (Adian Liusie, 2024; Li et al., 2023).

In this paper we present a solution to the order dependency problem. Our method Set-Based Prompting solves this problem by removing the ordering information from the inputs to the LLM, for a specified set of sub-sequences . We refer to this as prompting to emphasize that our method does not modify the underlying model, it only changes the input. These sub-sequences of multiple tokens are (from the model's perspective) run in parallel, leading to the output being agnostic as to the ordering of the sub-sequences 2 . We show that our method guarantees the parallel treatment of sub-sequences for arbitrary transformer based LLMs. We then test it on a variety of models on multiple choice question tasks and show that while parallel sub-sequences can impact performance, the impact is usually within that caused by re-ordering the sub-sequences when run without our method. We also discuss how the success of Set-Based Prompting suggests many future potential contributions.

2 Related Works

Multiple choice questions (MCQs) serve as an important task format in the evaluation of large language models (LLMs). Previous work has shown that modern LLMs are vulnerable to re-ordering of options in MCQs. Pezeshkpour et al., 2023 identify inherent 'positional bias' in the MCQ setting, where LLMs prefer to select options in specific positions as answers, and propose reordering the options to position the most likely candidates in specific locations within the sub-sequence. This is viable in the case where both the prior positional bias and the correct answers are known a priori, but not necessarily in more general settings. Zheng et al., 2024 identifies a slightly different problem of 'token bias', wherein LLMs prefer to select specific option IDs A/B/C/D as answers. They propose estimating prior token bias via a label-free, inference-time debiasing method, which separates the model's prior bias for option IDs from the overall prediction distribution. Adian Liusie, 2024 likewise identifies positional bias in the setting of pairwise comparisons of natural language generation, and suggests debiasing using estimates of prior positional bias derived from observing comparison outcomes for a subset of candidates. These methods all assume the existence of a set of comparable samples with which to estimate prior ordering biases. Other work looking at how modifying word choice affects the model's outputs (Salewski et al., 2024) show that this issue extends beyond order-dependency, but this matches how humans answer questions (Tjuatja et al., 2023) so should be less unintuitive for non-experts.

There is also an example of this order-dependence being used to probe the training data of LLMs. Oren et al., 2023 shows that the ordering of questions in the training data affects the ordering during inference, allowing for detection of training data contamination.

One implication of this MCQ order-dependence is that it makes the relative comparisons of the performance of different models on benchmarks such as MMLU less reliable, as different evaluation runs may employ different MCQ orderings within the test sets (Alzahrani et al., 2024).

3 Set-Based Prompting

To provably ensure order-independence we introduce Set-Based Prompting , an elegant 2-pronged modification to the model at inference time touching, respectively, the attention mask and the positional encoding. As these areas are not usually considered to be variable, this pushes the model slightly out of distribution. Furthermore, we hypothesize that some output quality degradation may occur since in the order-independent case there is both less computation and less information, as no tokens in the set of parallel sequences can attend to tokens in other parallel sequences. However, we hypothesize that base models are robust to minor changes, and output coherence will not be noticeably impaired. In fact, to see the impact of our methods requires many queries. We begin with a review of the attention mechanism's attention mask and the positional encoding system used in transformer-based LLMs. Then we will show how we can use a non-triangular mask combined, together with a modified positional encoding - our two prongs - to make the attention block unable to determine the order of sub-sequences. This results in completely removing all the ordering information between sub-sequences, making the model's outputs perfectly order invariant.

3.1 Background

Attention Mask. The attention mechanism is a means of taking in an arbitrary number of inputs and attending to them in a weighted manner. This allows for longer sequences to be input and processed into a single output. When processing text, the attention mechanism is used on each input token in turn, creating an embedding for each token separately.

In practice, we say the tokens can see all other tokens in the input, but when generating text the tokens in the input are usually treated as if they are the next to be generated and future tokens are masked output. This is seen in LLMs including GPT-2 (Radford et al., 2019) and Llama (Touvron, Lavril, et al., 2023) with their self-attention mechanism, wherein the outputs for a certain position in a sequence are based only on the known outputs at

Order-Dependant Prompting

Set-Based Prompting

j

=

1

2

3

4

5

6

7

j

=

1

2

3

2

3

4

5

i

= 1

the

1

0

0

0

0

0

0

i

= 1

the

1

0

0

0

0

0

0

i

= 2

aptly

1

1

0

0

0

0

0

i

= 2

aptly

1

1

0

0

0

0

0

i

= 3

quick

1

1

1

0

0

0

0

i

= 3

quick

1

1

1

0

0

0

0

i

= 4

light

1

1

1

1

0

0

0

i

= 2

light

1

0

0

1

0

0

0

i

= 5

reddy

1

1

1

1

1

0

0

i

= 3

reddy

1

0

0

1

1

0

0

i

= 6

1

1

1

1

1

1

0

i

= 4

1

0

0

1

1

1

0

i

= 7

fox

1

1

1

1

1

1

1

i

= 5

fox

1

1

1

1

1

1

1

p

(

i,

1)

p

(

i,

2)

p

(

i,

3)

p

(

i,

4)

p

(

i,

5)

p

(

i,

6)

p

(

i,

7)

p

(

i,

1)

p

(

i,

2)

p

(

i,

3)

p

(

i,

2)

p

(

i,

3)

p

(

i,

4)

p

(

i,

5)

+

+

+

+

+

+

+

+

+

+

+

+

+

+

X

= [

the

,

aptly

,

quick

,

light

,

reddy

,

,

fox

]

X

s

=

[

the

,

{

[

aptly

,

quick

]

,

[

light

,

reddy

,

]

}

,

fox

]

X

start

s

1

s

2

X

end

previous positions and not on future positions. This is implemented by multiplying the inputs by a mask vector that is 0 for all future tokens. When these mask vectors are stacked (for parallel computation), they form a triangular matrix, the attention mask matrix for that input. Figure 2 shows this attention mask for a sequence of 7 tokens.

Positional Encoding. Different language models use different methods of representing the position of input tokens within a sequence. The original transformer introduced by Vaswani et al., 2017 proposed absolute positional embeddings, which involves assigning a specific value vector at each time step, or position, and then adding element-wise those values to the token embedding. This is the positional encoding method used in models such as GPT-2 .

In contrast, most state of the art LLMs including lama2/3 , and PaLM (Anil et al., 2023), use rotary positional embeddings (RoPE) as introduced by Su, Lu, et al., 2023, which encode absolute positional information with a rotation matrix and naturally incorporate explicit relative position dependency in the self-attention formulation.

3.2 Provable Order-Independence via Set-Based Prompting

We now describe Set-Based Prompting , our technique for modifying the positional encoding to provide parallel (order-free) representation of sequences of text, and prove that it ensures order-independence. Set-Based Prompting can be applied to both absolute and RoPE embeddings, on both GPT-2 and Llama2/3 models. In the absolute embedding case, the same value is added to the token embedding for each token located in the same position in the parallel representation of the input text. Likewise in RoPE, absolute positional information is the same for two tokens in the same 'location' within parallel sequences, and a token within a parallel sequence retains relative positional dependency with other tokens within the same parallel sequence.

3.3 Methodology and Theoretical Guarantees

Consider the Attention block with positional encoding (Ferrando et al., 2024; Kobayashi et al., 2021; Su, Ahmed, et al., 2024) for single-headed attention 3 , the attention mechanism maps multiple vectors into a single vector in a differential way. We start with a fixed query vector x i which we give as an input column vector, X = [ x 1 , . . . , x n ] ∈ R d × n is the sequence of all n inputs that will be mapped to a single vector, and M is the attention mask, an n × n lower triangular matrix. The attention operator is defined as follows.

where W Q (query), W K (key), and W V (value) are d × d matrices, with d being the internal dimensions of the matrices. p ( i, j ) is the vector-valued positional encoding for the token at position j relative to position i ; note that if it is absolute (GPT-2) then p is only a function of j , while if relative (RoPE) it is a function of i -j .

Recall that x i is fixed. Equation (1) can therefore be considered to be a weighted sum over all columns of X , where the j th column is weighted by α i,j , and where M prevents future tokens from having any weight.

Since addition is commutative , the order in which the individual x j are provided is lost, except as provided by p ( i, j ) , which includes the order information both for the scaling ( α i,j ) and the unscaled vector ( x j + p ( i, j ) ). The attention mask M also functionally encodes positional information when attention blocks are applied in series since 'future' x j are masked out and do not contribute to the sum.

Thus, to prove that Set-Based Prompting is order-independent, it suffices to show that the positional encoding ( p ( i, j ) ) and attention mask ( M ) are unaffected by re-ordering by an arbitrary permutation of the parallel sub-sequences. Let S be the (unordered!) set of parallel sub-sequences, S = { s 1 , . . . , s /lscript } with ∑ l k =1 | s k | = n . We start by defining the indices on each token (subscript) and which sequences the tokens are members of (superscript), with x ∅ indicating a token is a member of no sub-sequences. The tokens before the parallel sub-sequences are treated identically as in the normal case, and we denote them by X start = [ x ∅ 1 , . . . , x ∅ r -1 ] where r is the lowest index in the parallel sub-sequences. The tokens after the parallel sub-sequences can be described similarly X end = [ x ∅ p +1 , . . . , x ∅ m ] where p is the greatest index in the parallel sub-sequences and m is the final index. Note that m = | X start | +( p -r ) + | X end | , where | · | is the length of a sequence of vectors.

Then, for a parallel sub-sequence s k we say that s k = [ x k r , . . . , x k q ] . Note that in our input representation there is no ordering on s k ∈ S , because we reset the subscripts (positional indices) at the beginning of each sub-sequence. We have simply indexed the sub-sequences for convenience.

By writing it this way we can directly input it into the ATTN( x i , X s , M ) function with X s = X start ◦ s 1 ◦ · · · ◦ s /lscript ◦ X end , where ◦ is concatenation. If we do this we will obtain order independence since p ( i, j ) and M i,j only use the indices of the tokens, so they are unaffected by ordering of the sub-sequences.

To show how p ( i, j ) and M i,j (the unmodified M ) are order independent we need to consider the three possible cases for the input vector x i : 1) x i is before the sub-sequences, 2) x i is in the sub-sequences, and 3) x i is after the sub-sequences. The first case is straightforward since x ∅ i ∈ X start . The output is unaffected by the ordering of S as all sub-sequence tokens are masked out by M i,j ; i < r .

The second case is where x k i ∈ s k with r ≤ i ≤ p . In this case if we naively evaluate ATTN( x k i , X s , M ) we will get a result that is unaffected by the ordering of S , but the activations will be further from the training distribution than necessary, resulting in lowered performance as discussed in Section 4.3. Note that for any x k j ∈ X s , p ( i, j ) and M ( i, j ) are only affected by the positional index j (instead of k ).

Finally, in the third case where x ∅ i ∈ X end , the same argument as the second case applies. The input vector x ∅ i can 'see' all previous tokens, but their positional indexing is unchanged under re-orderings of S . Thus in all three cases we have sub-sequence order independence.

3.3.1 Attention Mask

While the above method is sufficient for sub-sequence order independence, if used in practice the generated representations encountered by the learned weight matrices ( W V and ( W K ) will be far from the training distribution in both case 2 and case 3 encounter. In case 2 this is due to input vectors attending to all sub-sequences' inputs, e.g , if there were three sub-sequences starting at index i = 2 the first token of sub-sequence 1 ( x 1 2 ) would have non-zero weight on 4 tokens, three of which would have the positional encoding of 2 . In training, the model would never see multiple weights with the same positional encoding particularly multiple positional encodings that match the input token. Note that when used in an LLM ATTN() is applied sequentially, so even minor errors can be amplified. For case 3, the results are less severe since the positional encoding multiplicity only occurs for a set of previous tokens, it does not occur for the input token.

/negationslash

We can mitigate some of the issues by modifying the attention mask M ( i, j ) , making it so that case 2 tokens do not encounter out of distribution inputs. To do this we define a new attention mask M k,f i,j that also takes in the sub-sequences index each along with positional index of each token ( x k i , x f j , etc), while still retaining the sub-sequence order independence we want. With this notation, when k = f = ∅ we have that x k i and x f j are in the same parallel sub-sequence.

/negationslash

Equation 2 still maintains order independence since it only differs from M i,j when i = j and in that case does not depend on the values of i or j .

If we consider ATTN( x k i , X s , M k,f i,j ) we still retain order independence because ATTN( x k i , X s , M k,f i,j ) does not change under re-orderings of the labels, it only considers equality or absence of labels. See Section 4.3 for results when we don't do this masking.

This modified attention mask means that only tokens in case 3 see out of training distribution representations, and that no tokens will 'see' multiple other tokens at the same index.

This yields the following theorem.

Theorem 1 Given M k,f i,j as in Equation (2) , fix any permutation function τ on the indices 1 , . . . , /lscript of the sub-queries S = { s 1 , . . . , s k , . . . , s /lscript } for the attention mechanism, so that applying τ to the blocks of column vectors corresponding to the /lscript -th parallel sub-sequences transforms X s = X start ◦ { [ x 1 1 , ... ] , . . . , [ x k 1 , ... ] , . . . , [ x /lscript 1 , ... ] } ◦ X end to X ′ s = X start ◦ { [ x τ (1) 1 , ... ] , . . . , [ x τ ( k ) 1 , ... ] , . . . , [ x τ ( /lscript ) 1 , ... ] } ◦ X end . Then

and

4 Performance

While Set-Based Prompting is guaranteed to produce order independent results, we still need to test what other impacts it has on the generated text. To this end, we collected four different LLM families ( GPT-2 (Radford et al., 2019), Llama 2 (Touvron, Martin, et al., 2023),

OpenAI

Llama 2

Llama 3

Mistral

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst. 7B Inst.

Model

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Modified MMLU Accuracy

Ordering Effect

Worst of 2

Best of 1

Best of 2

Set-Based Prompting

OpenAI

Llama 2

Llama 3

Mistral

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst. 7B Inst.

Model

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Modified MMLU Accuracy

Ordering Effect

Worst of 2

Best of 1

Best of 2

Set-Based Prompting

(a) Modified CSQA results for each model

(b) Modified MMLU results for each model

Llama 3 (AI@Meta, 2024), and Mistral (Jiang et al., 2023)) to conduct our experiments (see figure 1 for the list of models). Due to our method modifying the internals of the model, each family requires a custom implementation of the inference code, meaning we can test all models in a family, but adding new families is time consuming. Our implementation for inputting parallel sub-sequences is implemented like the special tokens already commonly used by most LLMs, with three special 'tokens' ( start-parallel , new-sub-sequence , and end-parallel ) that are removed during inference, allowing for Set-Based Prompting to be added in existing workflows directly.

For testing we used two standard test sets CommonsenseQA (CSQA) (Talmor et al., 2019) and (Measuring Massive Multitask Language Understanding) (MMLU) (Hendrycks et al., 2020); these were chosen because they are multiple choice question sets that allow us to use the multiple options as the parallel sub-sequences.

As we want the runs with and without Set-Based Prompting to be similar we modified the prompting for both datasets. Instead of using numbers or letters before each option we quote the options in double quotes ( " ) and separate them with a space. This allows for the parallel queries to have the exact same set of tokens as the non-parallel queries. Of note, we implement the normal 1-shot evaluation for these tests that looks at the probability of each answer being generated, and we use the greatest value as the selected answer. We do observe that this slightly reduces the accuracy of the models from prompting with numbers or letters, so we label the methods 'modified CSQA' and 'modified MMLU' to differentiate our results. As we are concerned with the relative performance of our method, not in benchmarking the models, we believe this gives us results that are close to the existing literature.

4.1 CommonSenseQA

As we are concerned with reducing the variation of models under reordering, we examine both the accuracy under the default ordering, and the potential accuracy under reordering. In Figure 3a we examine our model's accuracy on CSQA, both with and without Set-Based Prompting . For the order dependent results we run CSQA twice and divide the answers into 4 sets (as in a confusion matrix): questions that both orderings get correct are counted and added to the Worst of 2 count, while questions that only one gets correct are also added to the Best of 2 bar, with the ones the normal ordering got correct used for Best of 1 . The counts are then divided by the total number of questions (9741) to give the mean accuracy. Thus, the range of the blue bars can be considered to be the possible range of accuracies under two possible orderings per question. Note that if the models understood the text the difference between orderings would be minuscule; the difference between just two orderings being large shows that the specific models are fundamentally incapable of solving multiple choice questions, they merely approximate a good response in expectation.

Llama 2

Llama 3

7B

7B Chat

13B

13B Chat

8B

0.15

0.20

0.25

0.30

0.35

0.40

Modified MMLU Accuracy

Set-Based Prompting

No

Yes

Llama 2

Llama 3

7B

7B Chat

13B

13B Chat

8B

8B Inst.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Modified CSQA Accuracy

Input Representation

Unmodified

Modified Positional Encoding

Modified Attention Mask

Set-Based Prompting

When we examine our method we see (1) that all ordering produce the same result, and (2) that the accuracy is within that of the variation of order dependent queries for all models except Llama 3, 8B Instruct . This result suggest that Set-Based Prompting is not adding additional information into the model, instead it is removing a dimension of variation. We discuss some hypotheses for how our method impacts the response generation in section 5.

4.2 Measuring Massive Multitask Language Understanding

Figure 3b shows the same analysis as in section 4.1 but for MMLU (Hendrycks et al., 2020). We see similar results to CSQA, but with lower accuracy on the more complex MMLU.

To further explore the impact of reordering on outputs we ran all possible permutations of MMLU option orderings through a subset of our models. In Figure 4 we see that for all Llama 2 models Set-Based Prompting is above or within the inter-quartile range of the 24 samples. We do see that in Llama 3 our method is as bad as the worst ordering, but it is still within the range of orderings.

4.3 Ablations

Partial Implementations. While we have a proof as to why both the the positional encoding and attention mask need to be modified, the transformer models are large enough that empirical observations are wanted to confirm the theory. We tested variations of Set-Based Prompting where only the attention mask or the positional encoding was modified to mask the positional information. Figure 5 shows the variation between normal ordering and reverse ordering of question options for the first 20 sets of questions in MMLU (ordered alphabetically). Interestingly, modifying the attention mask reduces variation much more than the positional encoding, although the effect varies significantly between models.

Enumerated Options. We ran a subset (first 20) of MMLU with the option numbers re-added, this lead to an improvement on all orderings. Note that adding numbers to inputs encoded with Set-Based Prompting implicitly removes the guarantee of order independence as there is an ordering implied by the numbers. See figure 10 in the appendix for this result.

Chain of Thought. We implemented a simple chain of thought prompt ('A: Let's think step by step') for the first 20 MMLU question sets. This lets us determine if the effects of Set-Based Prompting are limited to inputs near the end of the input sequence and check if an alternative prompting method would change the results. This method produced a moderate uplift in performance for both order dependent and order independent cases, note that this experiment was run once and not tuned. See figure 11 in the appendix for this result.

OpenAI

Llama 2

Llama 3

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst.

0.0

0.2

0.4

0.6

0.8

Sentiment Analysis Accuracy

Example Ordering

Worst of 2

Best of 1

Best of 2

Set-Based Prompting

Model

4.4 Other Tests

In Context Learning (Sentiment Analysis). We implemented a sentiment classification task with in context learning. The model was provided with 4 samples with labels, then asked to classify a fifth. The dataset is Financial Phrase Bank Malo et al., 2014 so all statements are finance related and classification is positive or negative. To look at the effects of ordering on the model performance we always had three samples with the same label and 1 with the other label, with the different label always being first or last. See figure 6a for the impacts of different orderings on the final classification task accuracy. Our experiment shows that Set-Based Prompting applied to the examples often improves performance over even the Best of 2 case.

Long-Context Information Retrieval. We implemented the long-context information retrieval task described in the paper by Liu et al. N. F. Liu et al., 2024. To do this, we generated 10 document sequences with the answer hidden in one of them. Then we moved where the answer was located from the first to the last document and observed the retrieval accuracy. We used the same templates and dataset as Liu et al. for this. To test the effects of Set-Based Prompting we ran the documents in parallel, either all 10, 5 groups (2,2,2,2), four groups (3,3,3,1) or two groups (5,5). When running the sets of documents in parallel there are two opposing forces affecting the accuracy: (1) parallelism, naturally, reduces order-dependence, which helps accuracy; (2) at the same time, the intervention moves the inputs farther out of distribution, reducing accuracy. Figure 6b shows the results of this experiment. Our experiment suggests that limited intervention is a sort of 'sweet spot'. The existence of a 'sweet spot' suggests that our method can be used to evaluate the robustness of the model's learned manifold, since we now have a subtle measure of model sensitivity to perturbations.

Extended Context Window. We checked if our method allows for the context window to be functionally extended since n , the total number of tokens, is less than m , the maximum positional index. Text generated where n was greater than the context window of the model maintains coherence when run, see the appendix section 6.12 for an example.

5 Discussion

Understanding why Set-Based Prompting works at all, without needing additional training, will require further research. We hope that adding Set-Based Prompting during training will reduce the impact on performance. As shown in Section 4, for all tested models Set-Based Prompting does not cause the model performance to degrade to an unusable state, although, as noted in Section 1, order dependency raises significant bias and reliability concerns.

We hypothesize three mechanisms of action for Set-Based Prompting to negatively impact an LLM's performance. First, Set-Based Prompting reduces the total amount of computation, thus reducing the model's ability to 'do work' leading to lower accuracy. Second, Set-Based

0

1

2

3

4

5

6

7

8

9

Position of Document with the Answer

0.4

0.5

0.6

0.7

# Documents

in Parallel

0

2

4

5

10

Prompting reduces the information available for some tokens, specifically those in the parallel sub-sequence, i.e., when the LLM is processing the second option it can see the first so can make a more nuanced embedding. Finally, Set-Based Prompting leads to out of distribution token embeddings. The out of distribution impact is suggested by the difference in impact on instruction tuned models, compared to their base models. For example, in Figure 3b we see that for Llama 3 , Set-Based Prompting performance is almost identical between the instruct and base models, while the instruct performs significantly better on best-of-2. This suggests that Set-Based Prompting is moving the input embeddings such that they are 'outside of' the fine-tuning, but the base model is robust enough to still understand them. These modifications to the model's output may also be mitigated by including examples of Set-Based Prompting inputs during training or fine-tuning, but additional work will be needed to determine this. Notably we also see hints of this when examining the model outputs under our methods; Set-Based Prompting makes fine-tuned models appear closer to the non-fine-tuned models, in particular Set-Based Prompting tends to make the fine-tuned models more verbose. We believe that each of these mechanisms can be tested, allowing for more details to be revealed about the inner workings of language models, and hopefully build towards reducing biases and increasing robustness of these systems.

5.1 Further Explorations

In this paper we present and evaluate a small change to the input representation. Our results suggest that larger changes may not only be possible, but may not require significant training overhead. We chose not to study fine-tuning as we believe that showing that our method is 'drop-in' is a significantly more powerful result than one that requires additional training. A next step would be to evaluate how to fine-tune models to be more receptive to this method, possibly even including the training as a part of base-model construction. Additionally, we chose to study the smallest possible version of this effect, a single parallel sub-sequence per prompt. Further study is merited to determine if more complex sub-sequence relationships can be input without significant degradation. In particular, we hypothesize that a version of this method that treats the input text as a directed graph of tokens (instead of a linear sequence) would be possible, i.e. , allow nested parallel sub-sequences, supporting cycles will likely require training. This is a representation already used for graphs (Ju et al., 2024; Wang et al., 2020) and is the natural representation for transformers. This type of representation would allow for inputs that much more closely represent how humans are provided text, e.g. , consider a class's syllabus, it is often a series of unordered sets of articles to read.Developing techniques that allow for reasoning about human behavior to be translatable to LLMs will greatly improve their utility to non-experts.

5.2 Towards Metadata in LLM Inputs

The empirical results in section 4.2 show that, while the positional encoding is used by LLMs, they can nonetheless 'understand' inputs where the positional encoding is modified. This suggests that LLMs are generally robust to minor changes in the token embeddings, with the positional encoding being the only currently implemented example. This is an entirely unexplored area of research. If we consider the positional encoding as a learned type of metadata, then other types of metadata could be added to tokens by simply adding new vectors during training to the token embeddings, just as we do with positional encodings. Adding additional metadata to each token would allow for 'out-of-band' signals that are added to input texts. Consider, for example, tagging each token with a vector that encodes the privilege level of the token, allowing for the implementation of instruction hierarchies (Wallace et al., 2024) that are encoded per token, instead of contextually by tags. Another, example of the utility of adding token metadata is in typography. Many digital texts have bold , underlines, italics , etc.; each of these could have an associated vector during training, allowing the LLM to 'see' the text more clearly. Instruction hierarchies and typography are just two possible uses of complex LLM inputs, and we believe that many more are possible, allowing for LLMs that can interact with the world's details and not just simple linear text.

As the usage of LLMs becomes more systemic, operators will need to be able to encode more complex representations, and have guarantees on the LLMs' behavior. We believe that this work presents a development towards that goal.

Acknowledgments

This research is funded in part by the Alfred P. Sloan Foundation , 'Pseudo-Randomness and the Crystal Ball,' (with O. Reingold), G-2020-13941, 2020, and NSF CAREER DMS-2340241.

References

Josh Achiam et al. 'Gpt-4 technical report'. In: arXiv preprint arXiv:2303.08774 (2023) (cit. on p. 1).

Mark J. F. Gales Adian Liusie Potsawee Manakul. LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models . 2024. arXiv: 2307.07889 [cs.CL] (cit. on pp. 2, 3).

AI@Meta. 'Llama 3 Model Card'. In: (2024). url : https://github.com/metallama/llama3/blob/main/MODEL_CARD.md (cit. on p. 7).

Norah Alzahrani et al. When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards . Feb. 1, 2024. arXiv: 2402.01781[cs] . url : http://arxiv.org/abs/2402.01781 (visited on 02/08/2024) (cit. on p. 3).

Shengnan An et al. 'Make Your LLM Fully Utilize the Context'. In: arXiv preprint arXiv:2404.16811 (2024) (cit. on p. 2).

Rohan Anil et al. 'Palm 2 technical report'. In: arXiv preprint arXiv:2305.10403 (2023) (cit. on p. 4).

Javier Ferrando et al. 'A Primer on the Inner Workings of Transformer-based Language Models'. In: arXiv preprint arXiv:2405.00208 (2024) (cit. on pp. 4, 14).

Dan Hendrycks et al. 'Measuring Massive Multitask Language Understanding'. In: International Conference on Learning Representations . 2020 (cit. on pp. 2, 7, 8).

Albert Q Jiang et al. 'Mistral 7B'. In: arXiv preprint arXiv:2310.06825 (2023) (cit. on p. 7).

Wei Ju et al. 'A comprehensive survey on deep graph representation learning'. In: Neural Networks (2024), p. 106207 (cit. on p. 10).

Nikita Kitaev et al. 'Reformer: The efficient transformer'. In: arXiv preprint arXiv:2001.04451 (2020) (cit. on p. 13).

Goro Kobayashi et al. 'Incorporating residual and normalization layers into analysis of masked language models'. In: arXiv preprint arXiv:2109.07152 (2021) (cit. on pp. 4, 13, 14).

Yingji Li et al. 'A survey on fairness in large language models'. In: arXiv preprint arXiv:2308.10149 (2023) (cit. on p. 2).

Nelson F Liu et al. 'Lost in the middle: How language models use long contexts'. In: Transactions of the Association for Computational Linguistics 12 (2024), pp. 157-173 (cit. on pp. 2, 9).

Pekka Malo et al. 'Good debt or bad debt: Detecting semantic orientations in economic texts'. In: Journal of the Association for Information Science and Technology 65.4 (2014), pp. 782-796 (cit. on p. 9).

R Thomas McCoy et al. 'Embers of autoregression: Understanding large language models through the problem they are trained to solve' . In: arXiv preprint arXiv:2309.13638 (2023) (cit. on p. 2).

Tsendsuren Munkhdalai et al. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention . 2024. arXiv: 2404.07143 [cs.CL] (cit. on p. 20).

Yonatan Oren et al. 'Proving test set contamination in black box language models'. In: arXiv preprint arXiv:2310.17623 (2023) (cit. on p. 3).

Niki Parmar et al. 'Image transformer'. In: International conference on machine learning . PMLR. 2018, pp. 4055-4064 (cit. on p. 13).

Pouya Pezeshkpour et al. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions . Aug. 22, 2023. doi : 10.48550/arXiv.2308.11483 . arXiv: 2308.11483[cs] . url : http://arxiv.org/abs/2308.11483 (visited on 04/03/2024) (cit. on pp. 2, 3).

Alec Radford et al. 'Language models are unsupervised multitask learners'. In: OpenAI blog 1.8 (2019), p. 9 (cit. on pp. 1, 3, 6).

Leonard Salewski et al. 'In-Context Impersonation Reveals Large Language Models' Strengths and Biases'. In: Advances in Neural Information Processing Systems 36 (2024) (cit. on p. 3).

Jianlin Su, Murtadha Ahmed, et al. 'Roformer: Enhanced transformer with rotary position embedding'. In: Neurocomputing 568 (2024), p. 127063 (cit. on pp. 4, 14).

Jianlin Su, Yu Lu, et al. RoFormer: Enhanced Transformer with Rotary Position Embedding . 2023. arXiv: 2104.09864 [cs.CL] (cit. on p. 4).

Richard Sutton. 'The bitter lesson'. In: Incomplete Ideas (blog) 13.1 (2019), p. 38 (cit. on p. 2).

Alon Talmor et al. 'CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge'. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . 2019, pp. 4149-4158 (cit. on p. 7).

Lindia Tjuatja et al. 'Do llms exhibit human-like response biases? a case study in survey design'. In: arXiv preprint arXiv:2311.04076 (2023) (cit. on p. 3).

Hugo Touvron, Thibaut Lavril, et al. 'Llama: Open and efficient foundation language models'. In: arXiv preprint arXiv:2302.13971 (2023) (cit. on p. 3).

Hugo Touvron, Louis Martin, et al. 'Llama 2: Open foundation and fine-tuned chat models'. In: arXiv preprint arXiv:2307.09288 (2023) (cit. on p. 6).

Dave Van Veen et al. 'Adapted large language models can outperform medical experts in clinical text summarization'. In: Nature Medicine (2024), pp. 1-9 (cit. on p. 2).

Ashish Vaswani et al. 'Attention is all you need'. In: Advances in neural information processing systems 30 (2017) (cit. on pp. 4, 13).

Eric Wallace et al. 'The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions'. In: arXiv preprint arXiv:2404.13208 (2024) (cit. on p. 10).

Guangtao Wang et al. 'Multi-hop attention graph neural network'. In: arXiv preprint arXiv:2009.14332 (2020) (cit. on p. 10).

Chujie Zheng et al. Large Language Models Are Not Robust Multiple Choice Selectors . Feb. 21, 2024. doi : 10.48550/arXiv.2309.03882 . arXiv: 2309.03882[cs] . url : http://arxiv.org/abs/2309.03882 (visited on 04/03/2024) (cit. on pp. 2, 3).

6 Appendix

6.1 Code Release

Full code for our experiments is available as a part of the final paper release. The code is released under the MIT license. The code can be found at this URL: https://github.com/reidmcy/set-based-prompting .

6.2 Compute Resources

Most of our results were done on a single Nvidia A100 GPU with 80GB of memory, running one model, thus the lack of 70B model results. Each run for table 3b took about 12 hours per model, with larger models taking slightly longer.

6.3 Computational Complexity

The computational complexity of our method is slightly lower than that of the normal attention mechanism. This is due to removal of elements from the attention mask, which reduces the number of key-value computations. This reduction is thus proportional to the number of parallel sequences and the length of these sequences. The reduction is thus minor for most uses, but could be significant for large numbers of parallel sequences or very long subsequences.

6.4 Full Attention Formulation

The formulation of attention presented in section 3.3 has the relevant components for this work, but does not represent the attention mechanism as used in most language models, since the development of multi-headed attention (Vaswani et al., 2017). Again this version is based on that presented in Kobayashi et al., 2021 with the attention mask made explicit and the vectors reordered to make the types work.

The attention mechanism with multiple heads, and bias is:

where the superscript h gives the index of the head, ◦ is the concatenation operator, and b h Q (query), b h K (key), and b h V (value) are the bias terms for each head. Note also that when used in a model the attention mechanism is likely to be used with a skip connection

and/or normalization. None of these listed changes affect the proof of presented in section 3.3, with the possible exception of the normalization which could in theory include positional information, but in all four model families we evaluated did not.

We also do not consider sparse attention mask patterns (Parmar et al., 2018), some of which are dependant on the token values (Kitaev et al., 2020).

6.5 Model Details

6.6 Alternative Proof

Below is an alternative proof of the main result using a different method. Note that the notation slightly differs from that used in the main proof

Consider the Attention block with positional encoding (Ferrando et al., 2024; Kobayashi et al., 2021; Su, Ahmed, et al., 2024) for multi-headed attention 4 . We take a sequence of n input vectors { x i } n i =1 ⊂ R D , written compactly as an input matrix X = [ x 1 ; . . . ; x n ] ∈ R D × n , where each x i is a column of X (also a token). For an input feature x j , we denote the feature matrix after incorporating the positional encoding matrix as X P,j = [ x p ( j, 1) 1 ; . . . ; x p ( j,n ) n ] .

The self-attention layer with L heads, attention mask matrix M , and parameters θ = { M , { ( Q m , K m , V m ) } L m =1 } (each M , Q m , K m and V m are D × D matrices) takes X as input and outputs a D × n matrix. In vector form, the j -th column is

where σ is the activation function, usually taken as the soft-max function.

In the standard decoder-only transformers, M is a lower triangle matrix with M j,j ′ = 1 for all j ≥ j ′ , and 0 otherwise. Such an M functionally encodes positional information when attention blocks are done in series since 'future' x j are masked out and do not contribute to the computation. In this case

which computes Attn θ ( X ) j based on the weighted summation of all previous tokens x j ′ for j ′ ∈ { 1 , 2 , . . . , j } , with weights given by σ ( 〈 Q m x j , K m x p ( j,j ′ ) j ′ 〉 ) .

We now propose to modify the matrix M to make the transformers order independent. Specifically, let us first assume there are l sub-sequences, with indices sets b 0 , b 1 , . . . , b l , b l +1 such that b 0 ∪ b 1 ∪ . . . ∪ b l +1 = [ n ] . Suppose we now want the model invariant with respect to the order of b 1 , . . . , b l , that is, we want the Attn θ ( X ) output to be unaffected by re-orderings of b 1 , . . . , b l in X .

To achieve this, our proposed Multidimensional Attention Mask sets M j,j ′ = 0 for all j ∈ b 1 ∪ . . . ∪ b l and j ′ / ∈ b λ ( j ) and j ′ ≤ j , where λ : [ n ] →{ 0 , 1 , . . . , l +1 } is the membership function such that λ ( j ) = k if and only if j ∈ b k . This makes sure that each sub-sequence cannot see other sub-sequences. In addition, we take positional encodings p ( j, j ′ ) that depend

only on the relative order between i and the rank of j within b λ ( j ) . As a result, the positional encoding for x j only depends on its rank within the subsequence x b j , and does not rely on other subsequences at all.

Let τ : R D × n → R D × n be an arbitrary permutation function that permutes the column blocks in X indexed by { b 1 , b 2 , ..., b l } . We have the following theorem.

Theorem 2 Suppose we define M as above. For any permutation function τ that permutes the columns blocks in X indexed by { b 1 , b 2 , ..., b l } . Attn θ ( X ) defined in (9) will be invariant to this order change:

This implies that for any j ∈ b l +1 ,

The proof of Theorem 2 can be found in Section 6.7 below. This theorem shows that by re-indexing the positional encoding and modifying the attention mask, the output of the attention block becomes independent of the ordering of the sub-sequences.

6.7 Additional Proof Details

Proof of Proposition 2:

As any permutation can be written as the composition of pairwise permutations, in the following, we prove the result for pairwise permutations. More specifically, suppose we consider a τ that permutes the columns indexed by b k and b ˜ k in X , and obtain ˜ X .

Recall that M is defined by letting M j,j ′ = 0 for all j ′ / ∈ b λ ( j ) and j ′ ≤ j . We then have for j ∈ [ n ]

In the following, we discuss three cases separately: j ∈ b k ∪ b ˜ k , j ∈ b 0 ∪ b l +1 , and j / ∈ b 0 ∪ b k ∪ b ˜ k ∪ b l +1 .

Case 1: j ∈ b k ∪ b ˜ k . Without loss of generality, we assume k < ˜ k . Then when j ∈ b k ,

implying

and the right-hand side equals to Attn θ ( τ ( X )) j .

When j ∈ b ˜ k ,

implying

and the right-hand side equals to Attn θ ( τ ( X )) j . Case 2: j ∈ b 0 ∪ b l +1 .

When j ∈ b 0 , we have

When j ∈ b l +1 , we have

Case 3: j / ∈ b 0 ∪ b k ∪ b ˜ k ∪ b l +1 . We have

implying

and the right-hand side equals to Attn θ ( τ ( X )) j .

OpenAI

Llama 2

Llama 3

Mistral

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst. 7B Inst.

0.0

0.1

0.2

0.3

0.4

0.5

Modified CSQA Accuracy

Question Ordering

Default Ordering

Set-Based Prompting

Reversed Ordering

Model

OpenAI

Llama 2

Llama 3

Mistral

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst. 7B Inst.

Model

0.0

0.1

0.2

0.3

0.4

Modified MMLU Accuracy

Question Ordering

Default Ordering

Set-Based Prompting

Reversed Ordering

6.8 Additional Plots

Figures 7 and 8 show the same data as 3a and 3b respectively, but with the different orderings accuracy's explicitly included. Figure 9 similarly shows the same data as figure 5.

6.9 Impact of Enumeration

Figure 10 shows the effects of adding numbers to the questions during input. Note, this is a removal of order independence for our method as the numbers implicitly encode positioning. We see that this improves performance across the board. We believe this is due to the models being trained with numbers on multiple choice questions.

OpenAI

Llama 2

Llama 3

7B

7B Chat

13B

13B Chat

8B

8B Inst.

Model

0.0

0.1

0.2

0.3

0.4

Modified MMLU Accuracy

Question Ordering

normal

normal_reversed

only_parallel_attention

only_parallel_attention_reversed

only_parallel_position

only_parallel_position_reversed

order_independent

OpenAI

Llama 2

Llama 3

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst.

Model

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Modified MMLU Accuracy

Ordering Effect

Worst of 2

Best of 1

Best of 2

Set-Based Prompting

6.10 Chain of Thought Prompting

6.11 Implementation

We conduct all experimental testing on huggingface models. We perform the greedy decoding with the temperature 0, and apply the zero-shot prompting in all experiments. These huggingface models already accept 2D attention masks as input to the forward() call, which accepts an attention mask and positional encoding as input and outputs a single generated output token. This 2D attention mask support is detailed in https://huggingface.co/blog/poedator/4d-masks. We make a few modifications to these model instance objects in order to enable model instances to accept 2D attention masks as input to the generate() call, which accepts an attention mask and positional encoding and outputs a sequence of generated output tokens. The modifications to Llama and Mistral model instances are identical.

A few additional minor modifications are required for GPT-2 model instances. On the GPT-2, Llama, and Mistral model architectures, we override the model instances' func-

OpenAI

Llama 2

Llama 3

GPT-2

7B

7B Chat

13B

13B Chat

8B

8B Inst.

0.0

0.1

0.2

0.3

0.4

0.5

Modified MMLU Accuracy

Ordering Effect

Worst of 2

Best of 1

Best of 2

Set-Based Prompting

Model

tion _update_model_kwargs_for_generation(). The function implementation remains the same, with the exception that if the function is called with a 2D attention mask, the 2D attention mask is converted to a 1D attention mask to be passed to subsequent forward calls. When generate() is called with a 2D attention mask, it passes that 2D mask to the first forward() call to generate the first token. After each call to forward, the function _update_model_kwargs_for_generation() defines the attention mask and position ids to be passed to the next forward() call. On second or later calls to forward, a 1D attention mask is passed, where attention is paid to all non-padding tokens in the input prompt and all tokens generated thus far. For the llama/mistral models, on subsequent calls, the subsequent position id is defined to consist of a vector containing only the next integer following the largest position id in the previous position id vector.

For example, suppose the input consists of the tokens A,B,C,D, where tokens B and C should be processed in parallel.

We would define the initial attention mask to be

and the initial position ids to be [1,2,2,3].

These values are passed to the first forward() call. On the next call to forward, we pass attention mask [1,1,1,1,1] and [[4]]. On the third call to forward, we pass attention mask [1,1,1,1,1,1] and position ids [[5]], and so on. The second generated token will attend to all original input tokens and the first generated token, and will utilize previously computed keys/values such that it views tokens B and C as though they were still processed in parallel.

On the GPT-2 instances, we override two additional functions. First, we override the '_attn' function of the GPTAttention object in each hidden layer. If a 2D attention mask is passed, then '_attn' skips generating a causal mask and uses the passed attention mask instead. This modification is unnecessary for Llama, which was already modified to pass a 2D attention mask through all the layers. Second, we override the function 'prepare_inputs_for_generation'. By default, this function regenerates position ids if both position ids and an attention mask are passed to generate(). Since we want to be able to pass custom position ids, we modify

the function to allow both the unmodified position ids and attention mask to be passed to subsequent forward calls.

6.12 Extending Context Window

One limitation of LLMs is their finite context windows. Existing work tries to circumvent this issue through techniques that require additional pretraining or and fine-tuning. In particular, Munkhdalai et al., 2024 proposes a new attention technique dubbed Infini-attention that scales LLMs to infinitely long inputs with bounded memory. This method, however, requires continual pre-training and fine-tuning, whereas we present Set-Based Prompting as a drop-in method of significantly increasing the context window when subsets of the text can be processed in parallel.

Due to Set-Based Prompting changing the positional encoding, inputs that exceed the trained context window of the model can be used since only the longest sub-sequence contributes to the positional encoding of the final input token. Thus, we can input longer sequences than the model was designed for. We tested this by inputting sequences of 6,000 tokens into Llama 2 7B, and the model was able to correctly summarize the input, while the same input put sequentially produces garbled text. See below for an example.

For a 6018 token input of the Wikipedia article for a recent disaster Llama 7B output:

When the text was broken up by header and fed in parallel the output was:

On March 26, 2024, the Francis Scott Key Bridge in Baltimore, Maryland collapsed after being struck by the container ship Dali. Two people were rescued from the water, and six maintenance workers were reported missing and presumed dead. The bridge was a continuous truss structure that was fracture critical, meaning it had no redundancy against failure of any particular part. The collision destroyed the southwest main truss pier, causing the central and south spans to collapse. The ship's emergency generators provided power to the propulsion system, but not to the electrical systems. The investigation is being led by the National Transportation Safety Board, with support from the Federal Bureau of Investigation, the U. S. Coast Guard, and the Maryland Department of Transportation. The ship's crew was accounted for, and no serious injuries were reported. The disaster has raised questions about the safety of larger ships passing

Further work should determine whether this drop-in extended context window method comes at the cost of output quality degradation, relative to methods like Infini-attention.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction are based on the results we present in the main body of the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We present a proof of our results including a discussion of the required assumptions for it to hold.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a proof of our main result in the paper.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Full code and datasets are provided with the paper allowing for complete reproduction of the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Full code and datasets are provided with the paper allowing for complete reproduction of the results.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper tests on a standard benchmark so the test details are well known and based on the literature.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Much of our results is a discussion of variability.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: While our main result is theoretical, we provide details on our testing compute resources in section 6.2 of the appendix.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our results do no use new data or human subjects, and we believe our method will reduce bias in LLMs.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work is motivated by observations of biases in LLMs, and we discuss how our method will impact these.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release a model or dataset.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the datasets used and respect their licenses.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is documented and released with the paper.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not have human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nState-of-the-art language models (LMs) are now used to perform tasks (such as, e.g., question answering) with no fine-tuning; these models are fed the question and target options in their context, and their next-token distribution is then used to select an answer from this set of options.\nLM outputs, however, are known to vary significantly based on the order in which these options are fed to it.\nThis paper proposes set-based prompting, a method to make LMs invariant to such option reorderings.\nIn short, the method feeds all options to a transformer model “in parallel” by modifying their position indexes to start from the same value, and by masking the attention so that options cannot attend to each other.\nIn practice, this makes the model run slightly out of distribution, as tokens presented to the model after the QA options will be able to attend to multiple tokens with the exact same position (as the multiple options are assigned this equivalent position indexes).\nThe paper has experiments on two tasks, question answering and language understanding.\n\nSTRENGTHS:\nThe paper presents a simple solution to a well known problem in NLP.\n\nThe paper runs experiments with a reasonable number of language model families: gpt-2, llama2, llama3, and mistral.\n\nWEAKNESSES:\nIn my opinion, this paper presents what is a relatively simple solution as being more complex than it is. (To be clear, I think this simplicity is one of the solution’s positive aspects, and not a negative aspect. Its presentation, however, could be considerably simpler.) Further, the proof of order invariance is sold as an important contribution, but it is also relatively straightforward. Since the methodological/theoretical contributions of this paper are relatively minor, though, I would expect it to have more experiments to confirm its efficacy. The paper, however, only presents two experiments, which are not very comprehensive in assessing the method's performance. More details below.\n\n* The paper’s contribution is a simple (yet potentially efficient) method to make LMs invariant to prompt ordering. This solution amounts to reindexing token positions and slightly changing the model’s attention mask. However, the paper takes almost 6 pages to present this relatively simple solution.\n* The paper only runs experiments on two tasks, and with one dataset each. This is not a comprehensive evaluation. The paper could run experiments with other tasks. Specifically, analysing the performance of this method on long-context information retrieval, as in Liu et al. (2024) could be interesting.\n* In-context learning is another important setting in which models are susceptible to input order variation. However, the paper only focuses its experiments on zero-shot experiments.\n\nLiu et al. (2024). Lost in the middle: How language models use long contexts\n\n\n\n\n------------\n\nEdit after author responses:\n\nI thank the authors for their response, and apologise for only replying to it after the discussion deadline. While I still think that some aspects of the paper could be significantly improved, e.g., simplifying the presentation, the newly added experiments make the paper's evaluation much more comprehensible, so I am raising my scores (from 3 to 5). The experiments showing the performance degradation on the long-context experiment are quite interesting (even if they present negative results), and experiments with ICL show another important setting for which this method can be useful.\n\nQUESTIONS:\nI don't have any specific questions. But as a suggestion, I think the stacked box plots in Figs 3 and 5 are quite hard to read. It would likely be easier to read them if the different conditions were presented as separate bars.\n\nLIMITATIONS:\nThe authors mention that their solution will make models run slightly out of distribution. Expanding on this point could be interesting, maybe discussing possible ways to mitigate this issue, such as fine-tuning models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of order dependency in Large Language Models (LLMs), which causes inconsistency in outputs when the order of semantically identical sub-sequences is changed. The authors propose a technique that eliminates order dependency in transformer-based LLMs. The method is both theoretically sound and experimentally validated, showing minimal impact on accuracy despite the inputs being out of distribution. The technique can be seamlessly integrated into fully trained models. The paper also explores the potential of enhancing LLMs by incorporating metadata information similar to positional encoding.\n\nSTRENGTHS:\n- The proposed method elegantly and effectively addresses the order dependency issue in LLMs with only a minor performance drop.\n- The method is supported by thorough experimental evaluation and theoretical guarantees, adding to its credibility.\n- The paper is well-written, clearly explaining the methodology and its implications.\n- Section 5.2 introduces an innovative idea of improving LLMs by using metadata information in a manner akin to positional encoding, which opens up exciting new avenues for research.\n\nWEAKNESSES:\nThe review does not identify any significant weaknesses in the paper. However, there are a few points of clarification needed, as outlined in the Questions section.\n\nQUESTIONS:\n- What is the impact of not enumerating the candidate answers (e.g., using A, B, C, D) versus simply using “”? This aspect does not appear to be addressed in the main text.\n- Regarding Figure 3, the proposed method seems to perform better than the worst-case scenario. However, if the goal is to achieve the highest score, it appears that the order-dependent method might still be preferable. Can the authors elaborate on this?\n- Why do the results differ in llama3? A more detailed explanation would help clarify this variation.\n\nThis paper presents a significant advancement in addressing order dependency in LLMs. The method is both theoretically and experimentally robust, providing a practical solution with minimal performance trade-offs. The clear writing and innovative approach, particularly in Section 5.2, add to the paper’s strengths. Addressing the questions and clarifying the identified points would further enhance the paper. Overall, this work is a valuable contribution to the field and is strongly recommended for acceptance.\n\n[edit: I increase my rating to Very Strong Accept, see my comment on author's rebuttal for justification]\n\nLIMITATIONS:\nLimitations are adressed in the paper", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to address an issue within large language models (LLMs): their sensitivity to the order of input sequences, known as order dependency. This problem causes LLMs to produce inconsistent outputs when the order of semantically identical inputs is changed. \n\n### Key Contributions:\n1. **Set-Based Prompting Technique**: The authors propose a novel technique called Set-Based Prompting, which ensures that the output of an LLM is invariant to the order of a specified set of sub-sequences. This method modifies the input representation by removing the order information from the inputs.\n  \n2. **Theoretical Guarantees**: The paper provides theoretical proofs demonstrating that Set-Based Prompting eliminates order dependency for any transformer-based LLM.\n\n3. **Empirical Evaluation**: The authors test their method on multiple LLMs, including GPT-2, Llama 2, Llama 3, and Mistral, using tasks like multiple-choice questions from the CommonSenseQA and MMLU datasets. They show that while Set-Based Prompting can slightly impact performance, it generally maintains accuracy within acceptable bounds and eliminates the variation caused by different orderings.\n------------------------------------------------------------------------\nThank you for your replies and I raised my score accordingly.\n\nSTRENGTHS:\n### Strengths\n\n- **Theoretical Rigor**: The paper includes rigorous theoretical proofs that validate the effectiveness of Set-Based Prompting in eliminating order dependency.\n- **Empirical Validation**: The extensive empirical evaluation on multiple models (GPT-2, Llama 2, Llama 3, and Mistral) on two datasets (CommonSenseQA and MMLU) demonstrates the robustness and applicability of the method.\n\nWEAKNESSES:\n### Weaknesses\n\n#### Implementation Complexity\n- **Engineering Trick Perception**: The method may be perceived as an engineering workaround rather than a fundamental advancement. It requires users to input sub-sequence information, which can be seen as an additional burden and may limit the method's practical applicability. Providing a more automated or integrated approach could make the technique more user-friendly.\n\n#### Limited Improvement in Performance\n- **Marginal Accuracy Gains**: While the method ensures order independence, the actual improvement in model performance (accuracy) is limited. The results show that Set-Based Prompting maintains accuracy within the variation caused by different orderings but does not significantly enhance it. Highlighting specific scenarios where this method offers substantial performance gains could strengthen the paper's impact.\n\n#### Dependency on User Input\n- **Manual Sub-Sequence Specification**: The requirement for users to specify sub-sequences manually can be a significant limitation. This dependency on user input reduces the method's usability and scalability, particularly for large datasets or applications where manual specification is impractical. Exploring ways to automate the identification of sub-sequences would be a valuable enhancement.\n\n#### Evaluation Scope\n- **Limited Dataset Variety**: The evaluation is primarily conducted on multiple-choice question datasets (CommonSenseQA and MMLU). While these are standard benchmarks, the scope of evaluation could be broadened.\n\n#### Theoretical vs. Practical Benefits\n- **Practical Utility of Theoretical Guarantees**: While the theoretical guarantees are robust, the practical benefits might not be as compelling if the accuracy remains within the range of existing variations. Providing more concrete examples or case studies where order independence significantly improves real-world applications could make the practical utility of the method more apparent.\n\n#### Misleading Paper Title\n- **Title Naming Concerns**: I do not think that the proposed method is kind of \"prompting\" paper. It does change the internal implementation of LLMs instead of simply \"prompting\" them.\n\nQUESTIONS:\nWhat will be the  benchmark performance under chain of thought prompting of this method compared to default chain-of-thought prompting baseline?\n\nLIMITATIONS:\nSee more details in Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}