Title: Lexinvariant Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Model

in-context learning

pretraining
Abstract: Token embeddings, a mapping from discrete lexical symbols to continuous vectors, are at the heart of any language model (LM). However, lexical symbol meanings can also be determined and even redefined by their structural role in a long context. In this paper, we ask: is it possible for a language model to be performant without \emph{any} fixed token embeddings? Such a language model would have to rely entirely on the co-occurence and repetition of tokens in the context rather than the \textit{a priori} identity of any token. To answer this, we study \textit{lexinvariant}language models that are invariant to lexical symbols and therefore do not need fixed token embeddings in practice. First, we prove that we can construct a lexinvariant LM to converge to the true language model at a uniform rate that is polynomial in terms of the context length, with a constant factor that is sublinear in the vocabulary size. Second, to build a lexinvariant LM, we simply encode tokens using random Gaussian vectors, such that each token maps to the same representation within each sequence but different representations across sequences. Empirically, we demonstrate that it can indeed attain perplexity comparable to that of a standard language model, given a sufficiently long context. We further explore two properties of the lexinvariant language models: First, given text generated from a substitution cipher of English, it implicitly implements Bayesian in-context deciphering and infers the mapping to the underlying real tokens with high accuracy. Second, it has on average 4X better accuracy over synthetic in-context reasoning tasks. Finally, we discuss regularizing standard language models towards lexinvariance and potential practical applications.

Lexinvariant Language Models

Abstract

Token embeddings, a mapping from discrete lexical symbols to continuous vectors, are at the heart of any language model (LM). However, lexical symbol meanings can also be determined and even redefined by their structural role in a long context. In this paper, we ask: is it possible for a language model to be performant without any fixed token embeddings? Such a language model would have to rely entirely on the co-occurence and repetition of tokens in the context rather than the a priori identity of any token. To answer this, we study lexinvariant language models that are invariant to lexical symbols and therefore do not need fixed token embeddings in practice. First, we prove that we can construct a lexinvariant LM to converge to the true language model at a uniform rate that is polynomial in terms of the context length, with a constant factor that is sublinear in the vocabulary size. Second, to build a lexinvariant LM, we simply encode tokens using random Gaussian vectors, such that each token maps to the same representation within each sequence but different representations across sequences. Empirically, we demonstrate that it can indeed attain perplexity comparable to that of a standard language model, given a sufficiently long context. We further explore two properties of the lexinvariant language models: First, given text generated from a substitution cipher of English, it implicitly implements Bayesian in-context deciphering and infers the mapping to the underlying real tokens with high accuracy. Second, it has on average 4X better accuracy over synthetic in-context reasoning tasks. Finally, we discuss regularizing standard language models towards lexinvariance and potential practical applications.

1 Introduction

All language processing systems rely on a stable lexicon, which assumes that a token (a word or subword such as tree ) has a consistent contribution to the meaning of a text (though of course this meaning is mediated by context). In neural language models (LMs), this contribution is the token embedding, which stably maps each token into a continuous vector [21, 16, 17, 7, 6]. However, in real language, a token's contribution might be determined by its structural role; in math and code, novel variable names are arbitrarily defined to carry new meaning, and poems such as Jabberwocky exploit humans' lexical flexibility in interpreting novel words such as vorpal . Besides standard language understanding, this lexical flexibility also correlates with a stronger in-context reasoning performance. For example, GPT-3 [6] and other large language models that demonstrate high lexical flexibility show strong performance on tasks involving in-context reasoning over new concepts and rules.

e

e _ c o p _ c e k e k

p('a big banana')

=

p('e cop cekeke')

=

e _ c o p _ c e k e k

(a) Lexinvariance

p('a big banana')

=

p('e cop cekeke')

=

Transformer

e _ c o p _ c e k e k

Embedding

e

Embedding

Embedding

e _ c o p _ c e k e k

Embedding

Embedding

Motivated by the above, we ask whether we can push this flexibility to the extreme: can we build a language model without any stable lexical mapping? To this end, we formulate and study such lexinvariant language models. We define a lexinvariant language model as a language model that assigns the same probability to all lexical permutations of a sequence. Formally, we define a lexical permutation π to be a one-to-one mapping of a set of lexical symbols 1 onto itself. Then the lexinvariant language model is defined as a language model over the symbol sequence x 1 ,...,x n with the following property:

For example, a lexinvariant language model (whose vocabulary is letters and space) should assign the same probability to the phrase ' a big banana ' as ' e cop cekeke ' because the two are the same up to the permutation π = { a : e , b : c , i : o , n : k , g : p , ···} (Figure 1a).

The central question is: how well can lexinvariant language models predict the next token given an increasingly long context? We find the answer is almost as well as standard language models, both theoretically and empirically. This is rather surprising given that lexinvariance seems like a strong limitation (a model doesn´ t know what any individual symbol means!) However, the intuition is that given longer contexts, a lexinvariant model can both infer the latent permutation π (lazily) up to whatever ambiguity is present in the language model, and do the standard next word prediction task jointly.

Theoretically, we prove that a constructed lexinvariant language model can converge to the true language model as the context length increases-that is, the average L1 distance between the predictions of the two models decreases with a convergence rate of O ( ( d T ) 1 4 ) , where T is the length of the context and d is the vocabulary size, and where the big-O notation hides polylogarithmic factors of d and T and an absolute constant that is independent of the language model.

Empirically, we train a lexinvariant LM by replacing standard embeddings in a decoder-only Transformer [24] with per-sequence random Gaussian vectors, such that the same symbols get the same embedding within each sequence but get different embedding across sequences (Figure 1b). We indeed see that the perplexity gap between the lexinvariant LM and the standard LM shrinks as context length increases, as shown in Section 3.2. With a 150M parameters Transformer and a small character-level vocabulary (130 tokens), the average perplexity gap shrinks from 9X to less than 1X the average perplexity of a standard LM after observing 512 tokens over The Pile [9]. With a larger 32K vocabulary, the gap also shrinks, especially on the more structured text like GitHub code, albeit at a much slower rate.

Wethen explore two additional properties of the lexinvariant LM: in-context deciphering and symbol manipulation. First, we show that given a ciphertext generated by applying a substitution cipher to English text, the lexinvariant LM can be seen as implicitly approximating Bayesian inference of the lexical permutation, i.e., cipher key, in-context. To show this empirically, we train a small MLP probe on top of a frozen pretrained lexinvariant LM to predict the deciphered token corresponding to the last

seen cipher token. We can then read out the inferred cipher key with each prefix of the sequence. We show that the accuracy of this inferred cipher key quickly improves as context length grows, reaching 99.6% average accuracy. We also show examples in Section 3.4 that visualize the uncertainties over different possible lexical mappings maintained by the lexinvariant LM when the cipher key is ambiguous and that the semantic meaning of a symbol with very rare occurrence can be inferred efficiently relative to other common symbols in context. Second, we show that lexinvariant models perform better than traditional models over synthetic pure in-context reasoning tasks that involve symbol manipulation. We observe a significant 4X improvement over a standard language model.

While the primary motivation of this paper is scientific exploration of a new idea, lexinvariance, we were also curious to see if it could help improve certain tasks, generalizing the performance gain we see on synthetic tasks. We stress that for most practical applications, lexinvariance is far too strong, so these experiments are intended to be illustrative rather than be a recipe for improving state-of-the-art. We discuss potential approaches to integrate the idea of lexinvariant LM into standard language modeling as a form of regularization, such that the LM assumes some form of partially stable symbol representations. The resulting LM can improve upon a standard language model over some BIG-bench tasks [23].

2 Lexinvariant Language Model

We define a language model as a probability distribution p ( x 1 ,...,x n ) over input token sequences x 1 ,...,x n ∈V n , where V is some vocabulary over symbols. A language model is lexinvariant if for all permutations π : V→V and for all token sequences x 1 ,...,x n ∈V n , p ( x 1 ,...,x n )= p ( π ( x 1 ) ,...,π ( x n )) . For example, if V = { a , b } , then the model should assign the same probability to aab and bba . One example p that satisfies this could simply be

Can such a lexinvariant language model predict language well, even though it can only make next token predictions based on the structure of co-occurence and repetition of input tokens in a single context?

2.1 Convergence on Language Modeling Performance

Weshow that we can construct a lexinvariant LM (as shown in Figure 2) to model the true language distribution faithfully, given a long enough context. The lexinvariant language model can essentially infer back the latent permutation π as it observes more symbols.

x

1

π

π

(x

1

)

x

2

x

3

x

n

π

(x

1

)

π

(x

1

)

π

(x

n

)

As an intuitive example, suppose that V = { a,b } and the true language only contains two sequences babbbb and ababab (and their prefixes) with even probability. When given only the first three letters, a lexinvariant model can't tell the latent permutation and can only assign the same probability to a and b for the next letter: Due to the lexinvariant property, it assigns the same probability to p ( a | aba )= p ( b | bab ) as well as to p ( b | aba )= p ( a | bab ) . Further, p ( a | aba )= p ( b | aba ) because the permutations to prefixes aba and bab are equally probable. In contrast, when considering the prefix abab , the fourth letter resolves the ambiguity in possible permutations π . (Since baba is not in the true language distribution, π cannot map a to b .) Therefore, the model can correctly predict that p ( a | abab )=1 and p ( b | abab )=0 .

Formally, for a given language model p , we define the associated lexinvariant language model p ′ ( x 1 ,...,x n ) as E π [ p ( π -1 ( x 1 ) ,...,π -1 ( x n ))] . Analyzing it, we have the following theorem:

Theorem 2.1. Let x 1 ,...,x n be any token sequence generated by an arbitrary language distribution p with an alphabet of size d . Let p ′ ( x 1 , ... , x n ) = E π [ p ( π -1 ( x 1 ) , ... , π -1 ( x n ))] . Then, for any

0 <ϵ,δ< 1 / 2 ,

with probability greater than 1 -δ , when T ≥ d ϵ 4 polylog( d, 1 ϵ , 1 δ ) , where the polylogarithmic term hides an absolute constant that is independent of p .

This theorem says that this associated lexinvariant language model converges to modeling the true language distribution fairly efficiently-with polynomial rate and near-linear dependence on vocabulary size d . Strikingly, this holds irrespective of the properties of the language distribution p 2 . In other words, a language model can indeed infer the operational meaning of the tokens in context based solely on the structure of the symbols !

Wegive a complete proof of this theorem in Appendix A. At a high level, this convergence happens because at most timesteps t , the new observation x t either provides new information about the permutation π , or x t has similar likelihood under the permutations that are likely given x 1 ,...,x t -1 . In the simplest case, if the posterior P ( π | x 1 ,...,x n ) concentrates on the correct π , then we converge to the standard LM. But even if it doesn't, that means the uncertainty about π should not matter for predicting the next token. We make this precise by interpreting p ′ ( x t | x 1 ,...,x t -1 ) as performing a multiplicative weights algorithm with the Hedge strategy of Freund and Schapire [8], and then relate the regret bounds to the average KL divergence between the predictions of p and p ′ , and ultimately the average L 1 distance between these predictions.

2.2 In-context Bayesian Deciphering

We can see the associated lexinvariant language model as implicitly learning to approximate an in-context Bayesian deciphering process, i.e. inferring a probability distribution over possible lexical permutations based on seen tokens, with the language modeling prior:

As shown above, p ′ can be reduced to two parts, where the first part is normal language modeling and the second part is the probability distribution of lexical permutations based on seen tokens. So the lexinvariant language model is implicitly learning to model P ( π | x 1 ,...,x n ) .

We can make this approximate in-context Bayesian deciphering explicit by training a small probe to predict P ( π | x 1 ,...,x n ) given the internal representation of the lexinvariant language model. We will show that this indeed recovers π reasonably accurately in the experiment section.

2.3 Constructing a Lexinvariant Language Model

Wenowconsider how to construct a lexinvariant LM in practice. A typical neural language model, such as a Transformer, converts input tokens to continuous vectors using token embedding and then passes these vectors as input to the rest of the neural network. Thus, the language model p it parameterizes depends on the token embedding E : V→ R d :

To make a neural LM lexinvariant, we can replace the standard stable token embedding E with a randomized E and take the expectation over E . Each token x ∈V has an independent embedding

E ( x ) ∼N (0 , I d ) , and the language model becomes

Since E d = E ◦ π , the right-hand side is the same when x i are applied with any permutation π , i.e., for any x 1 ,...,x n :

showing that the Transformer with random E is lexinvariant as in Eq. 1. Now we can train this lexinvariant LM similarly to a standard LM. Concretely, we sample a new E for each training sequence and minimizethestandardlanguagemodelinglossasinastandardneuralLM.Herewearestochasticallyoptimizing a variational lower bound of the standard language modeling loss with this randomized model by taking the expectation to the outside of the loss over log likelihood. Effectively, the same token gets the same random embedding within each training sequence, but different embedding across training sequences.

In practice, we focus on training decoder-only Transformers with a next token prediction objective in this work, where the model directly models p ( x n +1 | x 1 ,...,x n ) instead of the joint distribution. Our definitions and analysis above still hold in general. The only modification is that the final readout matrix also needs to be replaced with the same E , so that the Transformer can predict the embedding of the next token based on the embeddding of input tokens.

3 Experiments

3.1 Setup

Architecture. For all experiments, we use decoder-only Transformer architecture with T5 relative position bias [19]. We use models with 150M parameters, with 12 layers, 8 heads, head dimension 128, and MLP dimension 4096.

Training. Weuse the Adafactor optimizer [22], with a cosine decay learning rate schedule [13] from 0.01 to 0.001 based on preliminary experiments. We train the models from scratch for 250K steps on all the settings, with 512 sequence length and 64 batch size. We ran all of our experiments on 8 TPU cores. Our models are implemented in JAX [5].

Datasets. For datasets, we mainly use the Pile [9], a large open-source corpus that contains text collected from 22 diverse high-quality sources. We also run experiments on two additional datasets to explore their effects on the behavior of lexinvariant models: Wiki-40B [10], which contains high quality processed Wikipedia text in 40+ languages, and GitHub (subset of the Pile), which contains code from GitHub repositories with more than 100 stars and less than 1GB files.

3.2 Convergence to Standard Language Models

We first show empirically that lexinvariant LMs can mostly recover the next token prediction performance of standard LMs after a long enough context. As already discussed in section 2.1, the lexinvariant LM will theoretically converge to a standard LM as the context becomes long enough to resolve ambiguity. Here we verify this experimentally and show the variation of this convergence across corpora and tokenizations.

To show this, we train lexinvariant and standard LMs with both character-level vocabulary (128 ascii characters) and T5 default vocab (32k tokens) over the three datasets. For each model, we measure

0

100

200

300

400

Context Length

0

10

20

30

40

Perplexity

Model Perplexity with Character-level Vocabulary

standard

lexinvariant

0

100

200

300

400

Context Length

0

2000

4000

6000

8000

10000

12000

14000

Perplexity

Model Perplexity with T5 default vocab

standard

lexinvariant

the perplexity of each token in each sequence w.r.t. context length, smoothed by moving average within each sequence, i.e. P ( x i ,...,x i + k | x 1 ,...,x i ) 1 k for context length i . Weset the moving average window k =100 . Weplot results over 100 sequences. As shown in figure 3, the perplexity gap between lexinvariant LM and standard LM gradually shrinks as the prefix becomes longer and longer, albeit much more slowly with a larger vocabulary. This makes intuitive sense since a larger vocabulary has more possibilities of permutations and requires many more prefix tokens to disambiguate. For the 32K vocabulary, the 512 context length will only allow the model to see a very small number of tokens, let alone to see tokens more than once. Nonetheless, the model still manages to show the trend of convergence, since even a small number of repetitions can form common patterns in grammar (such as the usage of spaces, punctuation, articles, etc). For the character-level vocabulary, the perplexity gap shrinks from 9X to less than 1X the average perplexity of the standard LM. With a context length of 511, the lexinvariant LM converges to perplexity 3.38, almost comparable to the perplexity of the standard LMof2.00. Additionally, we observe that the gap shrinks significantly faster for models trained over Github than standard English text like Wiki-40B since code is more structured and it is easier to decipher the token permutation. We show the comparison across different datasets in Figure 7 in Appendix.

3.3 Recovering Substitution Ciphers

Here we show that lexinvariant LM is implicitly performing Bayesian in-context deciphering by testing its ability to recover cipher keys (e.g. Figure 4a) from character-level substitution ciphers, e.g. uC; kvR5W 4mfzd @f| Svcgn fw;m uCRmu;;d ]%/tildelow} :fBn . For the lexinvariant LM, this cipher text is perceived as the same as the quick brown fox jumps over thirteen lazy dogs , due to the lexinvariant property. It will then proceed to complete the cipher text with %d: uC; @f| with the same probability as it will complete the normal text with and the fox .

Because of this, we cannot directly read out the distribution of possible cipher keys P ( π | x 1 ,...,x n -1 ) implicitly inferred by the lexinvariant LM. To do this, we train a small two-layer MLP probe on top of a frozen trained lexinvariant LM. For each training sequence, we first embed the input sequence with a randomly sampled token embedding E as described in section 2.3 and obtain the hidden activation of the final layer generated by the frozen lexinvariant LM. Then, we pass this activation through the two-layer MLP probe. Finally, instead of decoding the output activations to classification logits with the same E as in the lexinvariant LM, we instead use another learnable non-randomized token embedding matrix E ′ so that the probe can recover the deciphered token with stable token embeddings. Overall, we train the probe jointly with this embedding matrix E ′ to predict the current token. Effectively, we are training the probe to decipher the current token using the representation provided by the lexinvariant LM. We train the probe over the same corpus as the original lexinvariant LMfor 10k steps. With this probe, we can directly visualize P ( π -1 ( x n ) | x 1 ,...,x n ) inferred by the lexinvariant LM, which is effectively one row in the permutation matrix representing π .

Nowwecanuse this probe to explicitly recover the cipher key. An example ground truth cipher key that we want to recover is shown in Figure 4a. Note that although the substitution cipher is only among lowercase letters, the character-level lexinvariant model we use assumes that all permutations among the 128 characters are possible , making the deciphering even more challenging.

%

5

4

:

@

;

B

R

C

S

]

W

c

f

d

g

m

k

n

v

u

w

|

z

}

~

%

5

4

:

@

;

B

R

C

S

]

W

c

f

d

g

m

k

n

v

u

w

|

z

}

~

a b c d e f g h

i

j

k

l mn o p q r s t u v w x y z

(b) Majority vote prediction

0

50

100

150

200

250

300

350

400

Context Length

0.0

0.2

0.4

0.6

0.8

1.0

(c) Cipher key prediction accuracy

a b c d e f g h

i

j

k

l mn o p q r s t u v w x y z

(a) Ground truth

%

4

5

:

;

@

B

C

R

S

W

]

c

d

f

g

k

m

n

u

v

w

z

|

}

~

%

4

5

:

;

@

B

C

R

S

W

]

c

d

f

g

k

m

n

u

v

w

z

|

}

~

%

4

5

:

;

@

B

C

R

S

W

]

c

d

f

g

k

m

n

u

v

w

z

|

}

~

%

4

5

:

;

@

B

C

R

S

W

]

c

d

f

g

k

m

n

u

v

w

z

|

}

~

%

4

5

:

;

@

B

C

R

S

W

]

c

d

f

g

k

m

n

u

v

w

z

|

}

~

a b c d e f g h

i

j

k

l mn o p q r s t u v w x y z a b c d e

f g h

i

j

k

l mn o p q r s t u v w x y z a b c d e

f g h

i

j

k

l mn o p q r s t u v w x y z a b c d e

f g h

i

j

k

l mn o p q r s t u v w x y z a b c d e

f g h

i

j

k

l mn o p q r s t u v w x y z

Concretely, we first input ciphertext through the frozen lexinvariant LM with the probe to produce a deciphered sequence. We then select a window of size 100 in the middle of the sequence and perform a majority vote over the corresponding deciphered tokens of each cipher token seen in this window. This essentially produces a predicted cipher key matrix for each window, and we can measure its precision against the ground truth. As shown in Figure 4c, such a cipher key prediction generally has increasingly higher precision as the window is selected later in the context, and it becomes near-perfect by the end of the sequence. Specifically, the cipher key matrix produced by the last window has an average precision of 99.6% over 1000 input sequences.

Finally, we aggregate over the last window of the 1000 sequences to recover a full cipher key, in case certain letters never appear in the last window of certain sequences. We again recover a full cipher key via majority vote. In Figure 4b, we show the highly accurate predicted cipher key recovered from ciphertext produced using the example ground truth cipher key in Figure 4a.

To perform a more detailed analysis showing the Bayesian deciphering process of the lexinvariant model, we use the logits of the probe to recover the predicted distribution of the cipher key P ( π | x 1 ,...,x n -1 ) . Instead of taking the majority vote of the predicted decipher tokens in the window, we take the mean of logits predicted for each ciphered token. This essentially gives a locally averaged predicted distribution of cipher key matrices. Specifically, the cipher key matrices are generated across windows of 50 characters, and the probabilities are averaged over 1000 input sequences encoded using the same ground truth cipher. As shown in Figure 5, the predicted distribution of cipher key matrix becomes sharper as the prefix becomes longer.

3.4 In-context Bayesian Deciphering Examples

Here, we show several qualitative examples of in-context Bayesian deciphering. We first show how the lexinvariant LM maintains uncertainty over possible lexical permutations while iteratively updating them at each index, using examples from a character-level lexinvariant model. Then, we also show an example of semantic in-context deciphering with a 32K vocabulary lexinvariant model, where the meaning of a novel word is inferred relative to common words in-context.

3.4.1 Uncertainty over Lexical Permutations

In Figure 6a, we input the following ciphered sequence to the frozen character-level lexinvariant LM with the probe: ' I saw lots of people in town today, walking and talking around me. I greeted my friend Alice and my classmate Alex. I saw a guy, Joe, walking outside carrying a zat. Joe's zat was taken off zy wind. Today's wind was strong, so Joe's zat flew zackward. Joe lost Joe's zat for good. Joe will miss Joe's zat. ' For each instance of z in the sequence, we display the predicted deciphering of that instance as a row of probabilities across non-cipher letters a -z .

The lexinvariant model starts off assuming uniform probability for all possible lexical permutations π . After seeing more and more text, the lexinvariant model quickly realizes that z only has a few main plausible decipherings ( b , h , c , m ). Eventually, the lexinvariant model is able to narrow the possibilities down to z maps to b near the end of the sequence. The predicted probabilities shift with the seen context accordingly, demonstrating an example of how the predicted cipher key is iteratively updated at each index.

Figure 6b shows another example with a similar set up, but with text: ' I saw a man in the pazk with a zat. The man was walking with the zat zight beside him. I've nevez seen anything like that befoze. ' While context initially suggests that z may be deciphered as c , it becomes clear that z must correspond to r after the appearance of ' right '. The disambiguation is reflected in the depicted probabilities.

In Figure 6c and 6d, we show two deciphering examples over code. We consider two code examples in which it is initially ambiguous whether the character z deciphers to : or { . The ambiguity is eventually resolved by the use of Python-like or Java-like syntax.

3.4.2 Semantic Deciphering

In addition to character-level deciphering, we show examples of semantic deciphering with the larger vocabulary of 32k. Although the lexinvariant LM could not possibly figure out the true lexical permutation among 32k tokens using a small 512 context, it is possible to construct a simple context that repetitively uses simple words so that these words are easier to decipher. Then the lexinvariant LM can decipher the approximate semantics of the rare symbols relative to other easier-to-decipher words.

One example is the following: given the prompt 'crash!' 'aaah!' i looked up from my cup of coffee. 'crash!' - that was the cafe window. and 'aaah!' [... more text...] what one here is a drink - restaurants - music - coffee - father the one here that drink is , where the word coffee , music , and father all only appear once before the question and restaurants appeared 4 times, the model is able to correctly answer that coffee is drinkable. See the full example in the appendix.

3.5 Synthetic Reasoning Tasks

As discussed in the introduction, lexical flexibility is correlated with in-context reasoning performance, as demonstrated by existing large LMs. Thus, we study whether the lexinvariant model also learns in-context reasoning capabilities through the challenging lexinvariant training.

Specifically, we measure the performance of lexinvariant models over two pure in-context symbol manipulation tasks: LookUp, where the task is to predict the next token based on the given lookup table, e.g. A->2 C->4 G->5 C-> (should predict 4 here); and Permutation, where the task is to permute an arbitrary subsequence of the given sequence the same way as in the given few demonstrations, e.g. A 2 C->C A 4 1 D-> (should predict D 4 here). In each of the tasks, the symbols are randomly sampled from the vocabulary so that we measure the pure reasoning ability independent from any knowledge of specific words. We measure the model performance in terms of generated token accuracy over 1000 examples. The results are shown in Table 1. As shown in the table, the lexinvariant models achieve drastically higher accuracy, with an average of 4X improvement.

Deciphering pred logits for cipher char 'z'

(b) True deciphering: ' z ' → ' r ', T =1 .

a b

c

d e

f

g h

i

j

k

l m n o p q

r

s

t

y

z

...outside carrying a

z

...a zat. Joe's

z

...was taken off

z

...strong, so Joe's

z

...Joe's zat flew

z

...Joe lost Joe's

z

...will miss Joe's

z

(a) True deciphering: '

z

'

→

'

b

',

T

=1

.

a

b

c

d

e

f

g

h

i

j

k

l m n o p q

r

s

t

u

y

z

...man in the pa

z

...pazk with a

z

...walking with the

z

...with the zat

z

...him. I've neve

z

...like that befo

z

Deciphering pred logits for cipher char 'z'

:

; <=> [ \

] ^ _ a b c d e f g h

i

j

k l mn o p q r s t u v w x y z { | }

binary_search()

z

...(high >= low)

z

...(arr[mid] > x)

z

...      } else

z

...}\n    } else

z

...void func2()

z

(c) True deciphering: '

z

'

→

'{',

T

=2

.

:

; <=> [ \

] ^ _ a b c d e f g h i

j

k l mn o p q r s t u v w x y z { | }

binary_search()

z

...(high >= low)

z

...(arr[mid] == x)

z

...(arr[mid] > x)

z

...        else

z

..._search()\n    else

z

...-1\ndef func2()

z

(d) True deciphering: '

z

'

→

':',

T

=3

.

3.6 Regularizing Language Models with Lexinvariance

Although lexinvariant LM has various interesting properties , it is not suitable for practical tasks since it would require the context to be extremely long so that all required words and knowledge are defined in the context. Here, we explore how to construct more practical semi-lexinvariant LMs that maintain some properties of lexinvariant LMs via regularization. We emphasize that this exploration is intended to be illustrative rather than directly improving state-of-the-art.

Instead of using random Gaussian embedding matrices in place of a learned embedding matrix entirely, we can use random embeddings for only some of the tokens in each sequence, while others use the learned embedding. This means that the learned LM assumes that certain tokens have stable meanings but not others, which can be seen as a form of regularization towards lexinvariance. Specifically, we randomly select tokens to randomize based on a Bernoulli distribution, which can essentially be seen as a form of dropout on token embeddings. On the BIG-bench tasks, we found that a model with dropout rate p =0 . 2 for randomization was 25% more likely to improve performance than to harm performance when evaluated with three shots, relative to a comparably-sized LM, with improvements especially over retrieval type of tasks. See full details in the Appendix G.

More broadly, this regularization view could potentially bring the benefit of lexinvariant LMs to practical applications. For example, the regularization could improve 1) the robustness of LMs by making them less sensitive to adversarial attacks or noise in the input data, 2) generalization across different languages or domains by being less tied to specific lexical items and more prone to learn the shared language structure, and 3) reasoning over more realistic tasks as we have started to explore with BIG-Bench. These areas are promising directions for future work to explore.

4 Related Work

4.1 Symbol Grounding

Beyondamodelingchoice, the main question of our paper (that being whether an LM can learn language without a stable token representation) is also analogous to the symbol grounding problem: Can meaning be acquired when symbols are not even grounded stably, i.e. they can be mapped to completely random meanings in different sequences? It has long been argued by the symbol grounding literature that symbolic representations must be grounded bottom-up in nonsymbolic representations [11], with famous arguments like Searle's Chinese room: It describes a person in a room given a step-by-step set of instructions by which they can respond to Chinese text with reasonable-sounding Chinese text. To an outside observer, the person in the room appears to understand Chinese, but the individual does not know a word of Chinese. This is widely used to argue that understanding language requires grounding the symbols in the real world. It leads to an ongoing debate on whether LMs can learn meaning purely from large amounts of text, without grounding to any real-world objects [4]. Although intuitively, lexinvariant LMsappears one step further removed from physical grounding than standard LM, we find that given enough context they can still infer the meaning of symbols based on lexical structures within the context.

4.2 Group invariances and Data augmentation

Our implementation of lexinvariant LMs can be seen as performing a form of very aggressive data augmentation, where we randomize the identity of each token in each sequence. From this perspective, it is somewhat similar to the data recombination in [14, 2] and augmentation of named entities in [20], where certain parts of the sentence are swapped with other words while still maintaining the original grammatical structure. In contrast to these augmentations, the training for our lexinvariant LMscompletely swaps out all parts of the input text.

4.3 Byte-level T5

There is existing work on absorbing tokenization completely into part of language modeling by using extremely small tokens, such as Byte-level T5 [25]. In the extreme, such a model would become closer and closer to lexinvariant LM, since bytes or bits have almost no stable meaning, so their embeddings are likely not used for prediction. In this paper, we study general lexinvariant LMs with the lexinvariant property baked in and without requiring specific tokenizers.

4.4 Deciphering Substitution Cipher using LMs

In general, solving substitution ciphers, where the cipher key is a permutation of the original alphabet, is a NP-hard problem when only having access to LMs that can assign probabilities to sequences [18]. There have been several works focusing on solving substitution ciphers using LMs, including approaches from searching over the permutation space guided by LMs' scores [12] to training a seq-to-seq model directly to perform deciphering as translation [3]. Although our work does not focus specifically on the task of deciphering substitution ciphers, we show that our lexinvariant model can efficiently perform in-context deciphering as a byproduct of language modeling.

4.5 Reasoning

It has been shown that large language models acquire surprising in-context reasoning capabilities [6, 15, 23]. Many of them are related to lexical flexibility through training for purely next-token prediction, such as modified arithmetic, data reformatting, and redefining single word etc. However, LLMsalso memorize an enormous amount of knowledge along the way, which is often unnecessary. This work can also be seen as an exploration of whether a (semi-)lexinvariant LM can discount knowledge and prioritize learning the diverse structural reasoning patterns in language, therefore achieving the strong reasoning capability of LLMs with a smaller model.

5 Conclusion

In this work, we define and study lexinvariant language models, which do not have stable embeddings and learn to infer the meaning of symbols in-context. We show several surprising properties of this model theoretically and empirically, including convergence to standard language modeling, in-context deciphering, and better reasoning capabilities. We also explore a less extreme lexinvariance regularized language model and demonstrate its potential for solving more practical tasks efficiently.

Acknowledgments and Disclosure of Funding

We thank Sang Michael Xie and Steven Cao for discussions and for providing feedback on our manuscript. This project is supported by Open Philanthropy Project Award. Qian Huang is supported by Open Philanthropy AI fellowship.

References

The multiplicative weights algorithm. https://www.cs.cmu.edu/afs/cs.cmu.edu/ academic/class/15859-f11/www/notes/lecture16.pdf . Accessed: 2023-04-24.

E. Akyürek, A. F. Akyurek, and J. Andreas. Learning to recombine and resample data for compositional generalization. ArXiv , abs/2010.03706, 2020.

N. Aldarrab and J. May. Can sequence-to-sequence models crack substitution ciphers? In Annual Meeting of the Association for Computational Linguistics , 2020.

E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , 2021.

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2019.

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. In European Conference on Computational Learning Theory , 1997.

L. Gao, S. R. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language modeling. ArXiv , abs/2101.00027, 2020.

M. Guo, Z. Dai, D. Vrandecic, and R. Al-Rfou. Wiki-40b: Multilingual language model dataset. In LREC2020 , 2020.

S. Harnad. Symbol grounding problem. Scholarpedia , 2:2373, 1990.

B. Hauer, R. B. Hayward, and G. Kondrak. Solving substitution ciphers with combined language models. In International Conference on Computational Linguistics , 2014.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv , abs/2203.15556, 2022.

R. Jia and P. Liang. Data recombination for neural semantic parsing. ArXiv , abs/1606.03622, 2016.

P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. R'e, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. J. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. S. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. F. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda. Holistic evaluation of language models. ArXiv , abs/2211.09110, 2022.

T. Mikolov, K. Chen, G. S. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In International Conference on Learning Representations , 2013.

T. Mikolov, M. Karafiát, L. Burget, J. Cernock` y, and S. Khudanpur. Recurrent neural network based language model. In Interspeech , 2010.

M. Nuhn and H. Ney. Decipherment complexity in 1:1 substitution ciphers. In Annual Meeting of the Association for Computational Linguistics , 2013.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(1), jun 2022.

J. Raiman and J. Miller. Globally normalized reader. In Conference on Empirical Methods in Natural Language Processing , 2017.

H. Schütze. Part-of-speech induction from scratch. In 31st Annual Meeting of the Association for Computational Linguistics , pages 251-258, Columbus, Ohio, USA, June 1993. Association for Computational Linguistics.

N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 4603-4611. PMLR, 2018.

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. A. Rahane, A. S. Iyer, A. Andreassen, A. Santilli, A. Stuhlmuller, A. M. Dai, A. D. La, A. K. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karakacs, B. R. Roberts, B. S. Loe, B. Zoph, B. Bojanowski, B. Ozyurt, B. Hedayatnia, B. Neyshabur, B. Inden, B. Stein, B. Ekmekci, B. Y. Lin, B. S. Howald, C. Diao, C. Dour, C. Stinson, C. Argueta, C. F. Ram'irez, C. Singh, C. Rathkopf, C. Meng, C. Baral, C. Wu, C. Callison-Burch, C. Waites, C. Voigt, C. D. Manning, C. Potts, C. T. Ramirez, C. Rivera, C. Siro, C. Raffel, C. Ashcraft, C. Garbacea, D. Sileo, D. H. Garrette, D. Hendrycks, D. Kilman, D. Roth, D. Freeman, D. Khashabi, D. Levy, D. Gonz'alez, D. Hernandez, D. Chen, D. Ippolito, D. Gilboa, D. Dohan, D. Drakard, D. Jurgens, D. Datta, D. Ganguli, D. Emelin, D. Kleyko, D. Yuret, D. Chen, D. Tam, D. Hupkes, D. Misra, D. Buzan, D. C. Mollo, D. Yang, D.-H. Lee, E. Shutova, E. D. Cubuk, E. Segal, E. Hagerman, E. Barnes, E. P. Donoway, E. Pavlick, E. Rodolà, E. F. Lam, E. Chu, E. Tang, E. Erdem, E. Chang, E. A. Chi, E. Dyer, E. J. Jerzak, E. Kim, E. E. Manyasi, E. Zheltonozhskii, F. Xia, F. Siar, F. Mart'inez-Plumed, F. Happ'e, F. Chollet, F. Rong, G. Mishra, G. I. Winata, G. de Melo, G. Kruszewski, G. Parascandolo, G. Mariani, G. Wang, G. Jaimovitch-L'opez, G. Betz, G. Gur-Ari, H. Galijasevic, H. S. Kim, H. Rashkin, H. Hajishirzi, H. Mehta, H. Bogar, H. Shevlin, H. Schütze, H. Yakura, H. Zhang, H. Wong, I. A.-S. Ng, I. Noble, J. Jumelet, J. Geissinger, J. Kernion, J. Hilton, J. Lee, J. F. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Koco'n, J. Thompson, J. Kaplan, J. Radom, J. N. Sohl-Dickstein, J. Phang, J. Wei, J. Yosinski, J. Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. Engel, J. O. Alabi, J. Xu, J. Song, J. Tang, J. W. Waweru, J. Burden, J. Miller, J. U. Balis, J. Berant, J. Frohberg, J. Rozen, J. Hernández-Orallo, J. Boudeman, J. Jones, J. B. Tenenbaum, J. S. Rule, J. Chua, K. Kanclerz, K. Livescu, K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. D. Dhole, K. Gimpel, K. O. Omondi, K. W. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang, L. Dugan, L. Qin, L. Contreras-Ochando, L.-P. Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He, L. O. Col'on, L. Metz, L. K. cSenel, M. Bosma, M. Sap, M. ter Hoeve, M. Andrea, M. S. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. Leavitt, M. Hagen, M. Schubert, M. Baitemirova, M. Arnaud, M. A. McElrath, M. A. Yee, M. Cohen, M. Gu, M. I. Ivanitskiy, M. Starritt, M. Strube, M. Swkedrowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, T. MukundVarma, N. Peng, N. Chi, N. Lee, N. G.-A. Krakover, N. Cameron, N. S. Roberts, N. Doiron, N. Nangia, N. Deckers, N. Muennighoff, N. S. Keskar, N. Iyer, N. Constant, N. Fiedel, N. Wen, O. Zhang, O. Agha, O. Elbaghdadi, O. Levy, O. Evans, P. A. M. Casares, P. Doshi, P. Fung, P. P. Liang, P. Vicol, P. Alipoormolabashi, P. Liao, P. Liang, P. W. Chang, P. Eckersley, P. M. Htut, P.-B. Hwang, P. Milkowski, P. S. Patil, P. Pezeshkpour, P. Oli, Q. Mei, Q. LYU, Q. Chen, R. Banjade, R. E. Rudolph, R. Gabriel, R. Habacker, R. R. Delgado, R. Millière, R. Garg, R. Barnes, R. A. Saurous, R. Arakawa, R. Raymaekers, R. Frank, R. Sikand, R. Novak, R. Sitelew, R. L. Bras, R. Liu, R. Jacobs, R. Zhang, R. Salakhutdinov, R. Chi, R. Lee, R. Stovall,

R. Teehan, R. Yang, S. J. Singh, S. M. Mohammad, S. Anand, S. Dillavou, S. Shleifer, S. Wiseman, S. Gruetter, S. Bowman, S. S. Schoenholz, S. Han, S. Kwatra, S. A. Rous, S. Ghazarian, S. Ghosh, S. Casey, S. Bischoff, S. Gehrmann, S. Schuster, S. Sadeghi, S. S. Hamdan, S. Zhou, S. Srivastava, S. Shi, S. Singh, S. Asaadi, S. S. Gu, S. Pachchigar, S. Toshniwal, S. Upadhyay, S. Debnath, S. Shakeri, S. Thormeyer, S. Melzi, S. Reddy, S. P. Makini, S. hwan Lee, S. B. Torene, S. Hatwar, S. Dehaene, S. Divic, S. Ermon, S. R. Biderman, S. C. Lin, S. Prasad, S. T. Piantadosi, S. M. Shieber, S. Misherghi, S. Kiritchenko, S. Mishra, T. Linzen, T. Schuster, T. Li, T. Yu, T. A. Ali, T. Hashimoto, T.-L. Wu, T. Desbordes, T. Rothschild, T. Phan, T. Wang, T. Nkinyili, T. Schick, T. N. Kornev, T. Telleen-Lawton, T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj, T. Khot, T. O. Shultz, U. Shaham, V. Misra, V. Demberg, V. Nyamai, V. Raunak, V. V. Ramasesh, V. U. Prabhu, V. Padmakumar, V. Srikumar, W. Fedus, W. Saunders, W. Zhang, W. Vossen, X. Ren, X. F. Tong, X. Wu, X. Shen, Y. Yaghoobzadeh, Y. Lakretz, Y. Song, Y. Bahri, Y. J. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou, Y. Bai, Z. Seid, Z. Xinran, Z. Zhao, Z. F. Wang, Z. J. Wang, Z. Wang, Z. Wu, S. Singh, and U. Shaham. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv , abs/2206.04615, 2022.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics , 10:291-306, 2021.

A Convergence Proof

Theorem A.1. Let x 1 ,...,x n be any token sequence generated by an arbitrary language distribution p with an alphabet of size d . Let p ′ ( x 1 , ... , x n ) = E π [ p ( π -1 ( x 1 ) , ... , π -1 ( x n ))] . Then, for any 0 <ϵ,δ< 1 / 2 ,

with probability greater than 1 -δ when T ≥ d ϵ 4 polylog( d, 1 ϵ , 1 δ ) .

Proof. For any desired error 0 <ϵ< 1 / 2 and failure rate 0 <δ< 1 / 2 , we will first prove the analogous statement for KL divergence instead of L 1 distance, and then relate a bound on KL divergence back to L 1 distance via Pinsker's inequality.

Throughout the rest of proof, we will work with a parameter ϵ ′ <O ( ϵ (log(1 /δ )) 1 / 4 ) < 1 2 , and will bound our KL divergence by ϵ ′ .

To prove the bound in terms of KL divergence, it will be useful to ensure to work with a 'smoothed' version of p , which we denote by ˜ p , for which every token has some nonzero probability, σ/d , of appearing at each timestep, for a parameter σ = δϵ ′ /T :

Similarly, let ˜ p ′ ( x 1 ,...,x n )= E π [˜ p -1 ( π ( x 1 ) ,...,π -1 ( x n ))] . Weuse ˜ P to denote the probabilities under this change. With probability at least 1 -σT ≥ 1 -ϵ ′ δ ≥ 1 -δ 2 , the realized sequence x 1 ,...,x n drawn under p can be regarded as being drawn from ˜ p (as these distributions can be coupled with this probability).

The key idea is then to show that ˜ p ′ ( y t +1 | y 1: t ) , where y t = π ∗ ( x t ) for some ground truth π ∗ unknown to p ′ , is equivalent to using the multiplicative weights algorithm to predict y t +1 with the Hedge strategy, with the experts being each possible permutation of the tokens and the cost incurred by each expert being the negative log likelihood of the prediction. We denote ˜ P π ′ ( y 1: n )= ˜ P ( y 1: n | π = π ′ )=˜ p ( π -1 ( y 1 ) ,...,π -1 ( y n )) and show this in Lemma A.2.

With this equivalence, we can then bound the difference between the prediction of p and p ′ as the regret of the multiplicative weights algorithm. Concretely, we show in Lemma A.3 that the regret of p ′ to any expert π is bounded as

for T ≥ ( 4log 2 ( d σ )log( d !) ) /ϵ ′ 4 .

Wecan see ˜ p as the particular expert/permutation ˜ P I . And we can further only consider the special case that π ∗ is also the identity permutation, then the same result holds over x t and with ˜ P π replaced by ˜ p , i.e.

Now we want to convert this bound on regret in terms of log likelihood to KL divergence, and eventually to L 1 distance. To convert it to KL divergence regret, we construct a martingale:

Weverify that this is a martingale in Lemma A.4, with differences bounded by 2log 1 σ , and bound the probability that Z T exceeds b =log d σ √ 8 T log 2 δ via Azuma's inequality Lemma A.6: with probability 1 -δ/ 2 , we have that | Z T | ≤ b .

Therefore, we have that with probability at least 1 -δ/ 2

Putting this all together, since 1 T ∑ T t log ˜ p i ( x t +1 | x 1: t ) ˜ p ′ ( x t +1 | x 1: t ) ≤ 2 ϵ ′ 2 for T ≥ ( 4log 2 ( d σ )log( d !) ) /ϵ ′ 4 , we have the following:

Wenowconvert our bound on KL divergence to a bound on L 1 distance via Pinsker's inequality:

Further, at any given x t , the difference between the redistributed probability distribution ˜ p and a unmodified probability distribution p is at most σ , so

Weare interested in the average L 1 across time steps:

where in the last inequality we applied Cauchy-Schwarz. Hence for T ≥ ( 4log 2 d σ log( d !) ) /ϵ ′ 4 ,

Simplifying this for

Wecan bound this average L 1 error by ϵ if we set ϵ ′ = ϵ 2 √ 2(2log 2 δ ) 1 / 4 < 1 2 , in which case our condition that T ≥ ( 4log 2 ( dT δϵ ′ )log( d !) ) /ϵ ′ 4 becomes T ≥ ( 512log 2 δ log 2 dT δϵ ′ log( d !) ) /ϵ 4 . The theorem now follows by simplifying this expression. Since log 2 δ ≤ 2log 1 δ , and log( d !) ≤ d log( d ) , we can relax the condition on T as

To remove the log 2 T from the right side, note that for any W > 10 , if T > 10 W log 2 W , then T >Wlog 2 T , yielding the further relaxed the condition on T as

Lemma A.2. Consider an arbitrary ground truth permutation π ∗ . For all time steps t ∈ [1 ,n ] , let y t = π ∗ ( x t ) . Consider the online prediction game of predicting y t +1 at each time step given previous observation y 1: t without knowing π ∗ but knowing ˜ p . Then, ˜ p ′ ( y t +1 | y 1: t ) is equivalent to the multiplicative weights algorithm's prediction of y t +1 with the Hedge strategy of Freund and Schapire [8], where it

Considers d ! experts corresponding to guessing each permutation π ′ is the ground truth permutation.

Maintains a weight w ( t ) π ′ for each expert at time step t , and the weights are initially as ˜ P ( π ) .

Picks a distribution across experts p ( t ) π ′ = w ( t ) π ′ Φ ( t ) where Φ ( t ) = ∑ j w ( t ) j .

Produces prediction of y t +1 as ∑ π ′ p ( t ) π ′ ˜ P π ′ ( y t +1 | y 1: t )

Receives a cost vector of m ( t ) π ′ = -1 ϵ log ˜ P π ′ ( y t +1 | y 1: t ) .

Updates the weights w ( t +1) i = w ( t ) i exp( -ϵm ( t ) i ) and repeat

Proof. Wecan first see that p ( t ) π ′ = ˜ P ( π ′ | y 1: t ) by induction:

Base case: p (0) π ′ = ˜ P ( π ) by assumption.

Inductive Case:

With the cost vector as m ( t -1) π ′ = -1 ϵ log ˜ P π ′ ( y t | y 1: t -1 ) , the update at step t is w ( t ) π ′ = w ( t -1) π ′ ˜ P π ′ ( y t | y 1: t -1 ) . Therefore, the probability over any particular expert π ′ is

This is equivalent to the update given by Bayes' rule when plugging in p ( t ) π ′ = ˜ P ( π ′ | y 1: t ) :

So we can conclude that p ( t ) π ′ = ˜ P ( π ′ | y 1: t ) , i.e. the process of updating the probability distribution across experts within the prediction game is equivalent to the process of the language model updating the probabilities ˜ P ( π ′ | y 1: t +1 ) across permutations π ′ . And this means that the algorithm's prediction ∑ π ′ p ( t ) π ′ ˜ P π ′ ( y t +1 | y 1: t )= ∑ π ′ ˜ P ( π ′ | y 1: t ) ˜ P π ′ ( y t +1 | y 1: t )= ˜ P ( y t +1 | y 1: t )=˜ p ′ ( y t +1 | y 1: t )

LemmaA.3. When using the Hedge strategy for the multiplicative weights algorithm, the average difference between the weighted distribution across experts and any particular expert π is bounded as

for ϵ ≤ 1 and for T ≥ ( 4log 2 ( d σ ) log( d !) ) /ϵ 4 .

Proof. Consider an arbitrary expert π .

We first show that the cost vectors are bounded by ρ = -1 ϵ log σ d : Recall we defined m ( t ) π = -1 ϵ log ˜ P π ( y t +1 | y 1: t ) . By the definition of our redistributed probability distribution, at time step t ∈ [1 ,...,T ] ,

By corollary 16.3 in [1], if we have cost vectors m ( t ) ∈ [ -ρ,ρ ] d ! , then for time T ≥ (4 ρ 2 log( d !)) /ϵ 2 where ϵ ≤ 1 ,

Note that we can simplify T ≥ ( 4log 2 ( d σ ) log( d !) ) /ϵ 4 .

Wecan now bound

By Jensen's inequality, we also have that

LemmaA.4. Let

Z i is a martingale.

Proof. Consider

Observe that Z i -1 has no dependence on x i +1 .

Therefore, Z i is a martingale.

LemmaA.5. | Z i -Z i -1 | ≤ c i where c i =2 | log d σ |

Proof. Wehave

In our redistributed probability distribution ˜ P , we have σ d ≤ ˜ P π ( x i | x 1: i -1 ) ≤ 1 for any π at any time i . Therefore,

Also, we can find an upper bound for the KL divergence by maximizing ˜ P I ( x i +1 | x 1: i ) to 1 and minimizing ˜ P ( x i +1 | x 1: i ) to σ d so that

We can maximize | Z i -Z i -1 | by maximizing the first term and minimizing the second term, or vice versa. In the first case, | Z i -Z i -1 | ≤ | log d σ -log σ d | = 2 | log d σ | . In the other case, | Z i -Z i -1 | ≤| 0 -log d σ | = | log d σ | .

Therefore, | Z i -Z i -1 | ≤ c i where c i =2 | log d σ |

Lemma A.6. By Azuma's inequality, with probability 1 -δ , we have that ∥ Z T ∥ ≤ b where b =2log d σ √ -8 T log 1 δ

Proof. By Azuma's inequality, for all positive reals b ,

Wecan rewrite in terms of δ =exp ( -b 2 8 ∑ T k =2 log 2 d σ ) so

B Model Architecture Details

In addition, we add a learnable scaling and bias parameter to the result of the embedding layer, so that the model can still learn to scale it as needed.

C Convergence on other datasets

Figure 7 shows the perplexity of lexinvariant LMs across the three different datasets. Note that Github converges significantly faster than standard Engish text like Wiki-40B, since code is more structured and easier to decipher the token permutation.

D Code Deciphering Full Examples

Therefore,

0

100

200

300

400

Context Length

10

0

10

1

10

2

10

3

10

4

Perplexity

pile,char

pile,32k

Wiki-40B,char

Wiki-40B,32k

github,char

github,32k

Python:

E Semantic Deciphering Full Example

'crash!' 'aaah!' i looked up from my cup of coffee. 'crash!' - that was the cafe window. and 'aaah!' - that was kate. people in the cafe shouted. kate and i ran to the window. there was no one there. then i turned to kate and put my arm around her. 'are you all right?' i asked. 'yes,' she said. 'i think so.' 'what is it?' some one shouted and a short red-faced man ran into the room. the man took my arm. 'matt! what are you doing to kate?' he asked. 'nothing, papa,' kate replied. 'it wasn't him. it was from out in the street.' the red-faced man looked at the window and then at me. he turned to his daughter. 'are you ok, kate?' he asked. kate gave him a little smile. 'yes, i think i am, papa,' she said. then her father spoke to me. 'sorry, matt. i heard kate and i thought...' 'that's ok, paolo,' i answered. it was ok. you see, this is soho, in the centre of london. in the day it's famous for music and films. at night people come and eat and drink

in the restaurants. expensive restaurants and cheap restaurants; italian restaurants and chinese restaurants. and day and night there are internet cafes like the web cafe. in soho you can buy any thing and any one. there are lots of nice people in soho. but there are also lots of people who are not very nice. i know because i live and work here. i often take a drink to a shop or cafe. i'm not rich and famous. and i don't know a lot. but i do know soho. what one here is a drink - restaurants - music - coffee - father the one here that drink is

Example prediction of the lexinvariant with 32k vocabulary train on the Pile:

The probability (at temperature=1) of coffee being selected is 56%, substantially higher than the next-highest probability of restaurant at 27%, music at 12%, or father at 5%.

F Synthetic Reasoning Task

Table 2: Synthetic Reasoning Tasks (adjusted for token frequencies)

G Language Models Regularized with Lexinvariance and BIG-bench Results

As described in the main paper, we implement a lexinvariance regularized Model in a way similar to embedding dropout. Note that one problem in implementing it naively by using random Gaussian embeddings and learned embedding in a mixture is that the two would become quickly distinguishable from each other during training since learned embeddings often have larger norms, allowing the model simply ignore the randomized tokens. So instead of using random Gaussian embedding matrices in place of a learned embedding matrix, we explored another approach for training a lexinvariant regularized LM: training a standard LM with learnable embedding matrix over sequences partially applied with a random token permutation B p ( x 1 ,π ) ,...,B p ( x 1 ,π ) , where B p ( x i ,π ) = π ( x i ) with probability p and B p ( x i ,π )= x i with probability 1 -p . Since each token can be remapped to any other token with equal chance, the produced model should ideally also be lexinvariant when p =1 , though with no strict guarantees. In practice, we found the models trained this way behave very similarly to models with random Gaussian embedding.

We evaluate our model over BIG-bench tasks where the language model performance scales well, and we prioritize evaluating generative tasks over multiple-choice tasks. Tasks we evaluated on:

gre reading comprehension.mul, linguistics puzzles.gen, linguistics puzzles.gen, rhyming.gen, tellmewhy.gen, simple arithmetic multiple targets json.gen, simple arithmetic json subtasks.gen, disfl qa.gen, arithmetic.gen, bridging anaphora resolution barqa.gen, matrixshapes.gen, sufficient information.gen, logical args.mul, novel concepts.mul, code line description.mul, unnatural in context learning.gen, unit interpretation.mul, english proverbs.mul, general knowledge.mul, geometric shapes.gen, human organs senses.mul, contextual parametric knowledge conflicts.gen, crass ai.mul,

auto categorization.gen, penguins in a table.gen, hindu knowledge.mul, english russian proverbs.mul, modified arithmetic.gen, cryobiology spanish.mul, evaluating information essentiality.mul, intent recognition.mul, understanding fables.mul, figure of speech detection.mul, empirical judgments.mul, simple ethical questions.mul, swahili english proverbs.mul, language identification.mul, phrase relatedness.mul, nonsense words grammar.mul, undo permutation.mul, object counting.gen, identify odd metaphor.mul, elementary math qa.mul, social iqa.mul, parsinlu qa.mul, metaphor understanding.mul, timedial.mul, causal judgment.mul, list functions.gen, implicatures.mul, date understanding.mul, codenames.gen, fact checker.mul, physics.mul, abstract narrative understanding.mul, emojis emotion prediction.mul, metaphor boolean.mul, strategyqa.gen, ascii word recognition.gen, auto debugging.gen, cause and effect.mul, conlang translation.gen, cryptonite.gen, cs algorithms.mul, dyck languages.mul, gender inclusive sentences german.gen, hindi question answering.gen, international phonetic alphabet transliterate.gen, irony identification.mul, logical fallacy detection.mul, movie dialog same or different.mul, operators.gen, paragraph segmentation.gen, parsinlu reading comprehension.gen, repeat copy logic.gen, rephrase.gen, simple arithmetic json.gen, simple arithmetic multiple targets json.gen, sports understanding.mul, word unscrambling.gen, hyperbaton.mul, linguistic mappings.gen, anachronisms.mul, indic cause and effect.mul, question selection.mul, hinglish toxicity.mul, snarks.mul, vitaminc fact verification.mul, international phonetic alphabet nli.mul, logic grid puzzle.mul, natural instructions.gen, entailed polarity.mul, list functions.gen, conceptual combinations.mul, goal step wikihow.mul, logical deduction.mul, conlang translation.gen, strange stories.mul, odd one out.mul, mult data wrangling.gen, temporal sequences.mul, analytic entailment.mul, disambiguation qa.mul, sentence ambiguity.mul, swedish to german proverbs.mul, logical sequence.mul, chess state tracking.gen, reasoning about colored objects.mul, implicit relations.mul, riddle sense.mul, physical intuition.mul, simple arithmetic json multiple choice.mul, geometric shapes.gen, gem.gen, simp turing concept.gen, common morpheme.mul, qa wikidata.gen, international phonetic alphabet transliterate.gen, similarities abstraction.gen, rephrase.gen, emoji movie.gen, qa wikidata.gen, word sorting.gen, emoji movie.gen, qa wikidata.gen, periodic elements.gen, hindi question answering.gen

Bellow, we plot the net percentage of tasks improved/deproved in each of the BIG-bench categories, out of the tasks that are changed by at least a threshold amount.

H Compute

Weuse one TPU v3-8 for all our pretraining runs. It takes approximately 23 hours for each pretraining run.

I Broader Impacts

Our work primarily provides a scientific exploration and understanding of the properties of lexinvariant language models. More broadly, these properties could potentially help improve the robustness, generalizability, and reasoning ability of LMs in the future works. In general we don't foresee more specific negative societal impacts from this work other than general misuse of language models.

Net Percent of Category Improved

40

20

40

Tasks

proved with a 1e-08% Threshold and 1 Shots

Net Percent of Category Improved

20

20

Tasks

Net Percent

1e-08% Threshold and 2 Shots

of Category Improved

40

19N

20

40

Tasks

Net Percent of Each Category Improved with a 1e-08% Threshold and 3 Shots

Net Percent of Category Improve

20

40

Task

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces lexinvariant language models, where the probability of any sequence is equal up to permutation of the vocabulary (e.g. P(A A B C) = P(B B C A)). In practice, the token embeddings within a Transformer model are replaced by random Gaussian vectors, which are consistent across a single sequence, but not over different examples. The authors show that a lexinvariant language model converges to modeling the true language distribution, in part because it is possible to infer the permutation (given a long enough sequence). While lexinvariant language models are not practical for many tasks, the paper shows that they can obtain better performance on some synthetic reasoning tasks. The authors also briefly discuss regularizing models with lexinvariance.\n\nSTRENGTHS:\nTo my knowledge, lexinvariance has not been explored before previously for natural language.\n\nThe experiments are pretty thorough, for example by demonstrating how lexinvariant models converge, exposing how to recover cipher permutations (with an additional probe), and showing superior performance on synthetic reasoning tasks where tokens have no predefined meaning.\n\nThe paper is generally clear and well-organized. It introduces the notion of lexinvariance, presents some theoretical properties (or characteristics) and describes how to construct a lexinvariant LM.\n\nThis paper challenges (to some degree) the notion that stable word/token embeddings are required for language modeling.\n\nWEAKNESSES:\nWhile the authors acknowledge the practical limitations of the approach, convergence can become quite slow as vocabulary size grows. \n\nGiven that the approach is mostly suitable for small vocabularies, a more thorough comparison with byte-level LMs (which also use a small vocabulary and tokens have limited semantics) could be helpful.\n\nWhile I find the paper interesting, the motivation behind the problem/approach is unclear.\n\nQUESTIONS:\nIs there a specific reason why spaces are not permuted for character-level examples? I would assume it's to simplify the presentation.\n\nHow long are the prompts for the synthetic reasoning tasks?\n\n[L45] Typo in doesnt\n\nLIMITATIONS:\nYes, limitations are discussed in the introduction (and elsewhere in the paper).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents “lexinvariant models” which eschew learning an embedding matrix for the symbols in the sequence and instead of modeling the identity of the symbols, they model the repetition and co-occurence patterns of the symbols in a sequence. Specifically, these symbols are interchangeable among all the items in the vocabulary and can be trained by drawing random vectors from a normal distribution to represent each symbol per sequence. This model is theoretically analyzed and it is shown that it can match with a standard autoregressive language model distribution as the context length increases. The models are mainly analyzed on symbol manipulation and decipherment tasks on synthetically generated data. A variant is also studied on natural language and code data.\n\nSTRENGTHS:\n– This is an interesting class of models and an intriguing question that investigates how far could we go with just the surface patterns of symbols and without knowing the symbols.\n\n– The experiments on synthetic data show that  such models have strong implicit decipherment capabilities and perform well on symbol manipulation tasks. This raises a possibility to augment standard language models with ideas from this approach to make the standard models more robust and performant on symbol manipulation and variable binding tasks.\n\n– The ideas are novel and the synthetic experiments are well-designed to explore the various facets of the models. The related work section does a good job of linking the ideas in this paper to other related topics in language modeling and symbolic representation.\n\nWEAKNESSES:\n– The experiments are mostly on synthetic data and tasks. The performance on real world data lags behind the simple standard approach. While interesting, the practical utility of these models is still questionable and their applicability feels limited to controlled toyish scenarios.\n\n– The theoretical analysis seems to have fairly loose bounds. Especially, the distribution matching looks feasible only for impractical long context lengths and there might be a large gap for realistic context lengths that the models and data typically work with today. Also, \\delta less than 0.5 is fairly loose.\n\n– Figure 2 seems awkward in the sense it implies that the model is essentially mixture of first order HMMs which seems far removed from the neural autoregressive models actually used in the experiments.\n\n– Section 3.2: 100 sequences seem too few as is suggested by the large variance of the proposed approach. Although the standard model seems to have a much smaller variance across the 100 sequences. This shows a further potential drawback of the proposed approach.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores neural language modelling without fixed token embeddings. Being invariant to symbols forces the model to rely on discovering patterns across sequences (e.g. co-occurrences, repeating symbols) that enable it to infer the underlying symbol sequence. For example, suppose the symbol-substituted sequence 123 123 145673 corresponds to “bye bye birdie”, the model can learn to predict the next symbol given enough training examples of different mappings of “bye bye” and, importantly, a sufficiently long context.\n\nThe authors start out by formally defining lexinvariance and proving its convergence to standard LMs. Their theorem establishes theoretically that lexinvariant LM performance converges to standard LM performance quite efficiently as context length grows. This paves the way to implement a lexinvariant LM. During training, each token is represented with random Gaussian embeddings. Identical tokens are mapped to the same embedding within a sequence but to different embeddings across different sequences. Results show that a character-level invariant LM converges to standard LM performance as context length grows, reflecting the convergence theorem empirically. It is also shown that the invariant LM is implicitly learning Bayesian deciphering over different potential symbol permutations, using probing to extract model ciphers. Finally, the authors introduce some potential practical uses of lexinvariance, such as improved performance on in-context symbol manipulation tasks and LM lexical regularisation.\n\nSTRENGTHS:\n-- #1 --\n\nThe theoretical and empirical findings of the paper are valuable contributions to our knowledge about the capabilities of language models and neural sequence models more generally. Fixed token embeddings are one of the core building blocks of LLMs and this paper explores LM behavior in their absence, leading to findings that many in the deep learning research community would find interesting.\n\n-- #2 --\n\nThe sections on symbol deciphering and uncertainty provide a useful analysis of how invariant LMs work under the hood. The roles of Bayesian deciphering and context length are made clear. The authors have proposed a model and explained a good deal about how it works. This is a nice contribution to the NLP interpretability literature.\n\n-- #3 --\n\nThe potential practical value of lexinvariant LMs (symbol manipulation tasks and lexical regularisation) are promising and worth further investigation. I could see this work being taken up by future researchers.\n\nWEAKNESSES:\n-- #1 --\n\nThe probing experiments can be improved by discussing some potential pitfalls of the technique in this context and/or including more results. The main pitfall to consider is the fact that the probe could be responsible for learning part of the cipher, as opposed to the lexinvariant LM. How do we separate what the lexinvariant LM has learned from what the probe learns by itself? In the extreme case of the lexinvariant LM not learning any deciphering (not realistic, just to demonstrate) the probe might still be able to learn some deciphering based on the contextual information in the LM representations. In the more realistic case, suppose the lexinvariant learns the entire cipher except for a few token mappings. Then the probe could infer the remaining mappings during its training. This is unlikely given the task complexity of deciphering, but it could be worth discussing. The paper would be strengthened by adding results that separate model knowledge from probe knowledge e.g. including cipher probe accuracy for a randomly initialised model.  See section 4.1 of this paper https://direct.mit.edu/coli/article/48/1/207/107571/Probing-Classifiers-Promises-Shortcomings-and for an overview of potential control methods.\n\n-- #2 --\n\nThe related works section is slightly disjointed. I understand that this is a result of the novelty of your idea and the far-reaching influences of the work. However, you could consider grouping the related work a bit more coherently. 4.1 and 4.3 could be connected as symbol grounding / invariance as augmentation, while 4.4 and 4.5 could be connected as downstream tasks related to your idea.\n* The symbol grounding subsection is interesting but requires slightly more explanation or references to previous works to clarify the matter to those unfamiliar with the problem. For example, Searle’s Chinese room argument is mentioned only by name.\n* The connection to ByT5 is not clear to me. In the extreme ByT5 would not become lexinvariant, it would just operate on finer grained units. Lexinvariance could be applied to any sequence unit or tokenisation scheme. If I’m misunderstanding something, please clarify, or perhaps consider omitting the subsection on ByT5.\n\nQUESTIONS:\n1. Include more examples of semantic deciphering. If possible, include demantic deciphering accuracy compared to a random multiple choice selection baseline. It’s clear from your current example that the model can figure out that has to copy one of the possible answers, but not clear if the correct prediction of “coffee” is down to anything other than chance.\n2. Could you please clarify the link you are making to ByT5?\n\n\n\n-- Typos --\n- Fix typo line 45 “doesn’t”.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the problem in which language model's tokens can be arbitrarily permuted (e.g., randomly interchanging two words $x$ and $y$ and stipulate that the new $x$ takes $y$'s meaning and vice versa -- such a process can happen over the entire vocabulary). The authors offer theoretical analysis and demonstrate that, to certain extent, it may be possible to train a language model invariant to permutations of the vocabulary.\n\nSTRENGTHS:\n- The paper provides a large set of interesting empirical experiments showcasing that\n- Despite the significant theoretical flaw (see below), the analysis provided in the paper is an interesting (though rather irrelevant) result with a novel proof idea. It is a pity that the derived result is not useful for the paper’s claim. The techniques used in the “proof” is quite creative as far as I can tell.\n\nWEAKNESSES:\nThe paper has an important theoretical flaw, which is that the constructed lexinvariant LM *may not* be an LM. Let’s consider a concrete example:\n\nIn the paper, the authors claim that, given an arbitrary LM $p(\\dotsm)$, one can construct a lexinvariant “LM” (call it $p'$) that is “almost equivalent” to $p$. The specific construction offered by the authors is $p'(x_1,\\dots,x_n)=\\mathbb{E}_{\\pi}[p(\\pi^{-1}(x_1),\\dots,\\pi^{-1}(x_n))]$. However, problems arise upon closer inspection. By this definition, we first have\n\n$$\np'(X_1=x_1,X_2=x_2)=\\frac{1}{2}p(X_1=x_1,X_2=x_2)+\\frac{1}{2}p(X_1=x_2,X_2=x_1).\n$$\n\nOn the other hand,\n\n$$\np'(X_1=x_1,X_2=x_2,X_3=x_3)=\\frac{1}{6}\\sum_{\\pi\\in S_3}p(X_1=x_{\\pi^{-1}(1)},X_2=x_{\\pi^{-1}(2)},X_3=x_{\\pi^{-1}(3)}).\n$$\n\nWhen we marginalize the above over $X_3$, we get\n\n$$\np'(X_1=x_1,X_2=x_2)=\\frac{1}{6}\\Big(\np(X_1=x_1,X_2=x_2)+\np(X_1=x_1,X_3=x_1)+\np(X_1=x_2,X_2=x_1)+\np(X_1=x_2,X_3=x_1)+\np(X_2=x_1,X_3=x_2)+\np(X_2=x_2,X_3=x_1)\n\\Big).\n$$\n\nWe can clearly see that, for most LM’s, the above doesn’t agree with the first marginal (unless they satisfy very special conditions)! This shows that, the constructed lexinvariant “LM” isn’t in fact an “LM”. It’s a sequence of incompatible joint distributions. In other words, one cannot construct a stochastic process based on the definition of $p'$ by invoking Kolmogorov’s extension theorem, because the marginals are incompatible.\n\nBased on the this, the paper’s various premises and conclusions start to fall apart.\n\nFirst of all, Theorem 2.1 becomes a “correct” but irrelevant result (it is slightly incorect in the sense that $p'$ is no longer a languge model). At best, Theorem 2.1 can be interpreted as that for an arbitrary LM, there exists a sequence of (incomptaible) conditional distributions such that, when $x_{1:T}$ is sampled from $p$, w.h.p., the (incompatible) conditionals are close to conditionals of $p$ (this is what appears to be what the proof is proving, though I have not checked the 6-page proof in the appendix too closely and can’t speak to its correctness). Such an understanding of this result would significantly undermine various claims in the paper in the following ways:\n\n1. First, one cannot conclude from the paper that given an arbitrary LM, there exists a “symmetrized” (i.e., lexinvariant) approximation of it. Therefore, one cannot, at least in theory, conclude that as the context gets longer, the “distance” between some lexinvariant model to an arbitrary LM will approach 0. \n    1. It is also quite likely that there exists such counterexamples — an LM in which no symmetric LM is a good approximation of it over any context length. If the authors are trying to provide a proof in that direction in the future, they may want to make an attempt at finding such counterexamples.\n    2. Furthermore, the empirical evidence is inconclusive as to the question of whether the specific implementation of the lexinvariant LM approximates the true conditionals. One may also look at Figure 3 and 7 and say that the perplexity curves have flattened out and that there is a systemic bias between the lexinvariant model and the standard language model.\n2. Next, since $p'(\\dotsm)$ is a sequence of incompatible joints, it is unclear if sampling from such a sequence of conditionals will yield something that can be justified.\n\nIf the authors cannot repair this theoretical flaw, I cannot recommend the paper accepted as-is since the paper’s current narrative revolves around a “theoretical proof then empirical validation” narrative. If the theory turns out to be irreparable, the paper should be rephrased as a series of experiments without theoretical underpinnings.\n\nFinally, a minor detail is that the presentation frequently switches notation for unclear reasons, which is very confusing. Most notably, it is unclear why the paper sometimes use $p$ but then immediately switches to $\\mathcal{P}$ (e.g., in Eq.(3), $\\mathcal{P}$ appears without introduction; similar situation happens in line 481).\n\nQUESTIONS:\n* In the proof in Appendix, A, line 482-483, can you elaborate why can we regard samples from $p$ and $\\tilde p$ as the same?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new language modelling approach, called *lexinvariance*,  where entries in vocabulary are permuted (with an unknown permutation). The paper proves that a lexinvariant language model can predict language well: it converges to a true language model with a polynomial rate in terms of context length. The paper presents empirical evidence to support that. Besides, lexinvariant LM can also infer unknown permutation (i.e. \"deciphering\"). It can also help improve LMs by being used as regularization (similar to dropout).\n\nSTRENGTHS:\n* The work is original. The idea of lexinvariance is interesting, and more importantly, the paper proves its convergence to true language model. Using gaussians to construct a lexivariant LM is a nice idea to reuse the standard LM architecture.\n\n* In terms of quality and clarity, the paper has a good balance between theory and experiment. The theoretical claim is proved, and supported by empirical evidence. The paper also shows possible applications of lexinvariance (deciphering, regularising a LM).\n\nWEAKNESSES:\nThe main weakness of the paper is that its application is limited. The most promising application is to use it to regularize a LM. However, it is unclear about the benefits of using lexivariant LM over using dropout. \n\nThe experimental setting for char-permutation maybe biased. We should also consider \"space\" as a char. Otherwise, the model would have some good hint for word tokenization.\n\nQUESTIONS:\n* It's confusing when using p as invariant LM (line 84) and a standard LM (the rest of the paper). \n\n* How to derive eq 3 from the definition of p'?\n\n* Line 314, \"lexinvariant LM is one step further away from physical grounding.\" This work hardly supports the claim, because it shows convergence of lexi LM to standard LM rather than to grounded LM.\n\nLIMITATIONS:\nIt is unclear about the work's application.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}