SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
T \geq \frac {d}{\epsilon^ {4}} p o l y l o g (d, \frac {1}{\epsilon}, \frac {1}{\delta}).
$$
Lemma A.2. Consider an arbitrary ground truth permutation  $\pi^{*}$ . For all time steps  $t\in [1,n]$ , let  $y_{t} = \pi^{*}(x_{t})$ . Consider the online prediction game of predicting  $y_{t + 1}$  at each time step given previous observation  $y_{1:t}$  without knowing  $\pi^{*}$  but knowing  $\tilde{p}$ . Then,  $\tilde{p}'(y_{t + 1}|y_{1:t})$  is equivalent to the multiplicative weights algorithm's prediction of  $y_{t + 1}$  with the Hedge strategy of Freund and Schapire [8], where it
- Considers  $d!$  experts corresponding to guessing each permutation  $\pi'$  is the ground truth permutation.
- Maintains a weight  $w_{\pi'}^{(t)}$  for each expert at time step  $t$ , and the weights are initially as  $\hat{\mathcal{P}}(\pi)$ .
- Picks a distribution across experts  $p_{\pi'}^{(t)} = \frac{w_{\pi'}^{(t)}}{\Phi^{(t)}}$  where  $\Phi^{(t)} = \sum_{j} w_{j}^{(t)}$ .
- Produces prediction of  $y_{t+1}$  as  $\sum_{\pi'} p_{\pi'}^{(t)} \tilde{\mathcal{P}}_{\pi'}(y_{t+1} | y_{1:t})$
- Receives a cost vector of  $m_{\pi'}^{(t)} = -\frac{1}{\epsilon} \log \tilde{\mathcal{P}}_{\pi'}(y_{t+1}|y_{1:t})$ .
- Updates the weights  $w_{i}^{(t + 1)} = w_{i}^{(t)}\exp (-\epsilon m_{i}^{(t)})$  and repeat
Proof. We can first see that  $p_{\pi'}^{(t)} = \tilde{\mathcal{P}}(\pi'|y_{1:t})$  by induction:
Base case:  $p_{\pi'}^{(0)} = \tilde{\mathcal{P}}(\pi)$  by assumption.
Inductive Case:
With the cost vector as  $m_{\pi'}^{(t-1)} = -\frac{1}{\epsilon} \log \tilde{\mathcal{P}}_{\pi'}(y_t | y_{1:t-1})$ , the update at step  $t$  is  $w_{\pi'}^{(t)} = w_{\pi'}^{(t-1)} \tilde{\mathcal{P}}_{\pi'}(y_t | y_{1:t-1})$ . Therefore, the probability over any particular expert  $\pi'$  is
$$
\begin{array}{l} p _ {\pi^ {\prime}} ^ {(t)} = \frac {w _ {\pi^ {\prime}} ^ {(t)}}{\Phi (t)} \\ = \frac {w _ {\pi^ {\prime}} ^ {(t - 1)} \tilde {\mathcal {P}} _ {\pi^ {\prime}} \left(y _ {t} \mid y _ {1 : t - 1}\right)}{\sum_ {j} w _ {j} ^ {(t - 1)} \tilde {\mathcal {P}} _ {j} \left(y _ {t} \mid y _ {1 : t - 1}\right)} \\ = \frac {p _ {\pi^ {\prime}} ^ {(t - 1)} \Phi^ {(t - 1)} \tilde {\mathcal {P}} _ {\pi^ {\prime}} \left(y _ {t} \mid y _ {1 : t - 1}\right)}{\sum_ {j} p _ {j} ^ {(t - 1)} \Phi^ {(t - 1)} \tilde {\mathcal {P}} _ {j} \left(y _ {t} \mid y _ {1 : t - 1}\right)} \\ = \frac {p _ {\pi^ {\prime}} ^ {(t - 1)} \tilde {\mathcal {P}} _ {\pi^ {\prime}} \left(y _ {t} \mid y _ {1 : t - 1}\right)}{\sum_ {j} p _ {j} ^ {(t - 1)} \tilde {\mathcal {P}} _ {j} \left(y _ {t} \mid y _ {1 : t - 1}\right)} \\ \end{array}
$$
This is equivalent to the update given by Bayes' rule when plugging in  $p_{\pi'}^{(t)} = \tilde{\mathcal{P}}(\pi'|y_{1:t})$ :
$$
\tilde {\mathcal {P}} \left(\pi^ {\prime} \mid y _ {1: t}\right) = \frac {\tilde {\mathcal {P}} \left(\pi^ {\prime} \mid y _ {1 : t - 1}\right) \tilde {\mathcal {P}} _ {\pi^ {\prime}} \left(y _ {t} \mid y _ {1 : t - 1}\right)}{\tilde {\mathcal {P}} \left(y _ {t} \mid y _ {1 : t - 1}\right)}
$$
So we can conclude that  $p_{\pi'}^{(t)} = \tilde{\mathcal{P}}(\pi'|y_{1:t})$ , i.e. the process of updating the probability distribution across experts within the prediction game is equivalent to the process of the language model updating the probabilities  $\tilde{\mathcal{P}}(\pi'|y_{1:t+1})$  across permutations  $\pi'$ . And this means that the algorithm's prediction  $\sum_{\pi'} p_{\pi'}^{(t)} \tilde{\mathcal{P}}_{\pi'}(y_{t+1}|y_{1:t}) = \sum_{\pi'} \tilde{\mathcal{P}}(\pi'|y_{1:t}) \tilde{\mathcal{P}}_{\pi'}(y_{t+1}|y_{1:t}) = \tilde{\mathcal{P}}(y_{t+1}|y_{1:t}) = \tilde{p}'(y_{t+1}|y_{1:t})$
Lemma A.3. When using the Hedge strategy for the multiplicative weights algorithm, the average difference between the weighted distribution across experts and any particular expert  $\pi$  is bounded as
$$
\frac {1}{T} \sum_ {t} ^ {T} \log \frac {\tilde {\mathcal {P}} _ {\pi} (y _ {t + 1} | y _ {1 : t})}{\tilde {p} ^ {\prime} (y _ {t + 1} | y _ {1 : t})} \leq 2 \epsilon^ {2}
$$
for  $\epsilon \leq 1$  and for  $T\geq \left(4\log^2\left(\frac{d}{\sigma}\right)\log (d!)\right) / \epsilon^4$
Proof. Consider an arbitrary expert  $\pi$ .
We first show that the cost vectors are bounded by  $\rho = -\frac{1}{\epsilon}\log \frac{\sigma}{d}$ : Recall we defined  $m_{\pi}^{(t)} = -\frac{1}{\epsilon}\log \tilde{\mathcal{P}}_{\pi}(y_{t + 1}|y_{1:t})$ . By the definition of our redistributed probability distribution, at time step  $t\in [1,\dots,T]$ ,
$$
\frac {\sigma}{d} \leq \tilde {\mathcal {P}} _ {\pi} (y _ {t + 1} | y _ {1: t}) \leq 1
$$
$$
\log \frac {\sigma}{d} \leq \log \tilde {\mathcal {P}} _ {\pi} (y _ {t + 1} | y _ {1: t}) \leq 0
$$
$$
0 \leq m _ {\pi} ^ {(t)} \leq - \frac {1}{\epsilon} \log \frac {\sigma}{d}
$$
$$
0 \leq m _ {\pi} ^ {(t)} \leq - \frac {1}{\epsilon} \log \frac {\sigma}{d}.
$$
By corollary 16.3 in [1], if we have cost vectors  $m^{(t)}\in [-\rho ,\rho ]^{d!}$  , then for time  $T\geq (4\rho^2\log (d!)) / \epsilon^2$  where  $\epsilon \leq 1$
$$
\frac {1}{T} \sum_ {t} ^ {T} p ^ {(t)} \cdot m ^ {(t)} \leq \frac {1}{T} \sum_ {t} ^ {T} m _ {\pi} ^ {(t)} + 2 \epsilon .
$$
Note that we can simplify  $T \geq \left(4\log^2\left(\frac{d}{\sigma}\right)\log (d!)\right) / \epsilon^4$ .
We can now bound
$$
\frac {1}{T} \sum_ {t} ^ {T} \left(p ^ {(t)} \cdot m ^ {(t)} - m _ {\pi} ^ {(t)}\right) \leq 2 \epsilon
$$
$$
\frac {1}{T} \sum_ {t} ^ {T} \left(\sum_ {\pi^ {\prime}} p _ {\pi^ {\prime}} ^ {(t)} m _ {\pi^ {\prime}} ^ {(t)} - m _ {\pi} ^ {(t)}\right) \leq 2 \epsilon
$$
$$
\frac {1}{T} \sum_ {t} ^ {T} \left(\sum_ {\pi^ {\prime}} \tilde {\mathcal {P}} \left(\pi^ {\prime} \mid y _ {1: t}\right) \left(- \frac {1}{\epsilon} \log \tilde {\mathcal {P}} _ {\pi^ {\prime}} \left(y _ {t + 1} \mid y _ {1: t}\right)\right) - \left(- \frac {1}{\epsilon} \log \tilde {\mathcal {P}} _ {\pi} \left(y _ {t + 1} \mid y _ {1: t}\right)\right)\right) \leq 2 \epsilon
$$
$$
\frac {1}{\epsilon T} \sum_ {t} ^ {T} \sum_ {\pi^ {\prime}} \left(\tilde {\mathcal {P}} \left(\pi^ {\prime} \mid y _ {1: t}\right) \left(\log \tilde {\mathcal {P}} _ {\pi} \left(y _ {t + 1} \mid y _ {1: t}\right) - \log \tilde {\mathcal {P}} _ {\pi^ {\prime}} \left(y _ {t + 1} \mid y _ {1: t}\right)\right)\right) \leq 2 \epsilon
$$
$$
\frac {1}{T} \sum_ {t} ^ {T} \mathbb {E} _ {\pi^ {\prime}} \log \frac {\tilde {\mathcal {P}} _ {\pi} (y _ {t + 1} | y _ {1 : t})}{\tilde {\mathcal {P}} _ {\pi^ {\prime}} (y _ {t + 1} | y _ {1 : t})} \leq 2 \epsilon^ {2}
$$
By Jensen's inequality, we also have that
$$
\frac {1}{T} \sum_ {t} ^ {T} \log \frac {\tilde {\mathcal {P}} _ {\pi} (y _ {t + 1} | y _ {1 : t})}{\mathbb {E} _ {\pi^ {\prime}} \tilde {\mathcal {P}} _ {\pi^ {\prime}} (y _ {t + 1} | y _ {1 : t})} \leq 2 \epsilon^ {2}
$$
Lemma A.4. Let
$$
Z _ {i} = \sum_ {t = 1} ^ {i} \left(D _ {K L} (\tilde {\mathcal {P}} _ {I} (x _ {t + 1} | x _ {1: t}) \| \tilde {\mathcal {P}} (x _ {t + 1} | x _ {1: t})) - \log \frac {\tilde {\mathcal {P}} _ {I} (x _ {t + 1} | x _ {1 : t})}{\tilde {\mathcal {P}} (x _ {t + 1} | x _ {1 : t})}\right)
$$
$Z_{i}$  is a martingale.
$$
\begin{array}{l} \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} [ Z _ {i} ] = \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} \left[ \sum_ {t = 1} ^ {i} \left(D _ {K L} (\tilde {\mathcal {P}} _ {I} (x _ {t + 1} | x _ {1: t}) \| \tilde {\mathcal {P}} (x _ {t + 1} | x _ {1: t})) - \log \frac {\tilde {\mathcal {P}} _ {I} (x _ {t + 1} | x _ {1 : t})}{\tilde {\mathcal {P}} (x _ {t + 1} | x _ {1 : t})}\right) \right] \\ = \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} \left[ D _ {K L} \left(\tilde {\mathcal {P}} _ {I} \left(x _ {i + 1} \mid x _ {1: i}\right) \| \tilde {\mathcal {P}} \left(x _ {i + 1} \mid x _ {1: i}\right)\right) - \log \frac {\tilde {\mathcal {P}} _ {I} \left(x _ {i + 1} \mid x _ {1 : i}\right)}{\tilde {\mathcal {P}} \left(x _ {i + 1} \mid x _ {1 : i}\right)} + Z _ {i - 1} \right] \\ \end{array}
$$
Observe that  $Z_{i - 1}$  has no dependence on  $x_{i + 1}$ .
$$
\begin{array}{l} \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} [ Z _ {i} ] = \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} \left[ \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} \log \frac {\tilde {\mathcal {P}} _ {I} (x _ {i + 1} | x _ {1 : i})}{\tilde {\mathcal {P}} (x _ {i + 1} | x _ {1 : i})} \right] - \mathbb {E} _ {x _ {i + 1} \sim \tilde {\mathcal {P}} _ {I}} \left[ \log \frac {\tilde {\mathcal {P}} _ {I} (x _ {i + 1} | x _ {1 : i})}{\tilde {\mathcal {P}} (x _ {i + 1} | x _ {1 : i})} \right] + Z _ {i - 1} \\ = Z _ {i - 1} \\ \end{array}
$$
Therefore,  $Z_{i}$  is a martingale.
Lemma A.5.  $|Z_{i} - Z_{i-1}| \leq c_{i}$  where  $c_{i} = 2|\log \frac{d}{\sigma}|$
$$
| Z _ {i} - Z _ {i - 1} | = \left| D _ {K L} (\tilde {\mathcal {P}} _ {I} (x _ {i + 1} | x _ {1: i}) \| \tilde {\mathcal {P}} (x _ {i + 1} | x _ {1: i})) - \log \frac {\tilde {\mathcal {P}} _ {I} (x _ {i + 1} | x _ {1 : i})}{\tilde {\mathcal {P}} (x _ {i + 1} | x _ {1 : i})} \right|
$$
In our redistributed probability distribution  $\tilde{\mathcal{P}}$ , we have  $\frac{\sigma}{d} \leq \tilde{\mathcal{P}}_{\pi}(x_i | x_{1:i-1}) \leq 1$  for any  $\pi$  at any time  $i$ . Therefore,
$$
\log \frac {\sigma}{d} \leq \log \frac {\tilde {\mathcal {P}} _ {I} (x _ {i + 1} | x _ {1 : i})}{\tilde {\mathcal {P}} (x _ {i + 1} | x _ {1 : i})} \leq \log \frac {d}{\sigma}.
$$
Also, we can find an upper bound for the KL divergence by maximizing  $\tilde{\mathcal{P}}_I(x_{i + 1}|x_{1:i})$  to 1 and minimizing  $\tilde{\mathcal{P}} (x_{i + 1}|x_{1:i})$  to  $\frac{\sigma}{d}$  so that
$$
\begin{array}{l} D _ {K L} \left(\tilde {\mathcal {P}} _ {I} \left(x _ {i + 1} \mid x _ {1: i}\right) \| \tilde {\mathcal {P}} \left(x _ {i + 1} \mid x _ {1: i}\right)\right) = \sum_ {x _ {i + 1}} \tilde {\mathcal {P}} _ {I} \left(x _ {i + 1} \mid x _ {1: i}\right) \log \frac {\tilde {\mathcal {P}} _ {I} \left(x _ {i + 1} \mid x _ {1 : i}\right)}{\tilde {\mathcal {P}} \left(x _ {i + 1} \mid x _ {1 : i}\right)} \\ \leq \log \frac {d}{\sigma} \\ \end{array}
$$
We can maximize  $|Z_{i} - Z_{i-1}|$  by maximizing the first term and minimizing the second term, or vice versa. In the first case,  $|Z_{i} - Z_{i-1}| \leq |\log \frac{d}{\sigma} - \log \frac{\sigma}{d}| = 2|\log \frac{d}{\sigma}|$ . In the other case,  $|Z_{i} - Z_{i-1}| \leq |0 - \log \frac{d}{\sigma}| = |\log \frac{d}{\sigma}|$ .
Therefore,  $|Z_{i} - Z_{i - 1}|\leq c_{i}$  where  $c_{i} = 2|\log \frac{d}{\sigma} |$
Lemma A.6. By Azuma's inequality, with probability  $1 - \delta$ , we have that  $\|Z_T\| \leq b$  where  $b = 2\log \frac{d}{\sigma}\sqrt{-8T\log\frac{1}{\delta}}$
Proof. By Azuma's inequality, for all positive reals  $b$ ,
$$
\begin{array}{l} P \left(Z _ {T} - Z _ {1} \geq b\right) \leq \exp \left(\frac {- b ^ {2}}{2 \sum_ {k = 2} ^ {T} c _ {k} ^ {2}}\right) \\ P \left(Z _ {T} - Z _ {1} \leq b\right) \geq 1 - \exp \left(\frac {- b ^ {2}}{2 \sum_ {k = 2} ^ {T} c _ {k} ^ {2}}\right) \\ \geq 1 - \exp \left(\frac {- b ^ {2}}{8 \sum_ {k = 2} ^ {T} \log^ {2} \frac {d}{\sigma}}\right) \\ \end{array}
$$
We can rewrite in terms of  $\delta = \exp \left(\frac{-b^2}{8\sum_{k=2}^{T}\log^2\frac{d}{\sigma}}\right)$  so
$$
\begin{array}{l} b = \sqrt {- \left(8 \sum_ {k = 2} ^ {T} \log^ {2} \frac {d}{\sigma}\right) \log \delta} \\ \leq \log \frac {d}{\sigma} \sqrt {- 8 T \log \frac {1}{\delta}} \\ \end{array}
$$
$$
P \left(Z _ {T} - Z _ {1} \leq b\right) \geq 1 - \delta
$$
B Model Architecture Details
In addition, we add a learnable scaling and bias parameter to the result of the embedding layer, so that the model can still learn to scale it as needed.
C Convergence on other datasets
Figure 7 shows the perplexity of lexinvariant LMs across the three different datasets. Note that Github converges significantly faster than standard English text like Wiki-40B, since code is more structured and easier to decipher the token permutation.
D Code Deciphering Full Examples
binary_search()z if (high >= low)z mid = (high + low) / 2; if (arr[mid] == x) return mid; if (arr[mid] > x) z high = mid - 1; return binary_search(); } else z low = mid + 1; return binary_search(); } } else z return -1; } } void func2()z
Figure 7: Smoothed Token Perplexity over the Pile, Wiki-40B and Github, with character-level and T5 default vocab
Python:
binary_search()z if (high  $> =$  low)z mid  $=$  (high  $^+$  low）//2 if( arr[mid]  $\equiv = x)\mathbf{z}$  return mid if( arr[mid]  $\rightharpoondown$  x)z high  $=$  mid-1 return binary_search() elsez low  $=$  mid  $+1$  return binary_search() elsez return -1 def func2()z
E Semantic Deciphering Full Example
'crash!' 'aaah!' i looked up from my cup of coffee. 'crash!' - that was the cafe window. and 'aaah!' - that was kate. people in the cafe shouted. kate and i ran to the window. there was no one there. then i turned to kate and put my arm around her. 'are you all right?' i asked. 'yes,' she said. 'i think so.' 'what is it?' some one shouted and a short red-faced man ran into the room. the man took my arm. 'matt! what are you doing to kate?' he asked. 'nothing, papa,' kate replied. 'it wasn't him. it was from out in the street.' the red-faced man looked at the window and then at me. he turned to his daughter. 'are you ok, kate?' he asked. kate gave him a little smile. 'yes, i think i am, papa,' she said. then her father spoke to me. 'sorry, matt. i heard kate and i thought...' 'that's ok, paolo,' i answered. it was ok. you see, this is soho, in the centre of london. in the day it's famous for music and films. at night people come and eat and drink
in the restaurants. expensive restaurants and cheap restaurants; italian restaurants and chinese restaurants. and day and night there are internet cafes like the web cafe. in soho you can buy any thing and any one. there are lots of nice people in soho. but there are also lots of people who are not very nice. i know because i live and work here. i often take a drink to a shop or cafe. i'm not rich and famous. and i don't know a lot. but i do know soho. what one here is a drink - restaurants - music - coffee - father the one here that drink is
Example prediction of the lexinvariant with 32k vocabulary train on the Pile:
- coffee. and i
The probability (at temperature  $= 1$ ) of coffee being selected is  $56\%$ , substantially higher than the next-highest probability of restaurant at  $27\%$ , music at  $12\%$ , or father at  $5\%$ .
F Synthetic Reasoning Task
Table 2 shows a variant of the synthetic reasoning task results in Subsection 1, where the symbols are instead sampled proportion to the token frequencies. Although the improvement still generally holds, the standard LM with character-based vocabulary becomes significantly better. We believe that this is because the model can get a significant advantage by guessing among the most common letter.
Dataset	Vocab	LookUp Acc	Permutation Acc
Standard	LI	Standard	LI
Pile	char	72.80	90.95	40.63	60.47
32k	61.20	90.95	40.55	54.55
Wiki-40B	char	75.55	63.45	42.71	59.86
32k	41.05	57.95	26.81	51.86
Github	char	66.00	86.75	36.62	70.77
32k	59.25	78.45	37.46	65.04
Table 2: Synthetic Reasoning Tasks (adjusted for token frequencies)
G Language Models Regularized with Lexinvariance and BIG-bench Results
As described in the main paper, we implement a lexinvariance regularized Model in a way similar to embedding dropout. Note that one problem in implementing it naively by using random Gaussian embeddings and learned embedding in a mixture is that the two would become quickly distinguishable from each other during training since learned embeddings often have larger norms, allowing the model simply ignore the randomized tokens. So instead of using random Gaussian embedding matrices in place of a learned embedding matrix, we explored another approach for training a lexinvariant regularized LM: training a standard LM with learnable embedding matrix over sequences partially applied with a random token permutation  $B_{p}(x_{1},\pi),\ldots ,B_{p}(x_{1},\pi)$ , where  $B_{p}(x_{i},\pi) = \pi (x_{i})$  with probability  $p$  and  $B_{p}(x_{i},\pi) = x_{i}$  with probability  $1 - p$ . Since each token can be remapped to any other token with equal chance, the produced model should ideally also be lexinvariant when  $p = 1$ , though with no strict guarantees. In practice, we found the models trained this way behave very similarly to models with random Gaussian embedding.
We evaluate our model over BIG-bench tasks where the language model performance scales well, and we prioritize evaluating generative tasks over multiple-choice tasks. Tasks we evaluated on:
gre reading comprehension.mul, linguistics puzzles.gen, linguistics puzzles.gen, rhyming.gen, tellmewhy.gen, simple arithmetic multiple targets json.gen, simple arithmetic json subtasks.gen, disfl qa.gen, arithmetic.gen, bridging anaphora resolution barqa.gen, matrixshapes.gen, sufficient information.gen, logical args.mul, novel concepts.mul, code line description.mul, unnatural in context learning.gen, unit interpretation.mul, english proverbs.mul, general knowledge.mul, geometric shapes.gen, human organs senses.mul, contextual parametric knowledge conflicts.gen, crass ai.mul,
auto categorization.gen, penguins in a table.gen, hindu knowledge.mul, english russian proverbs.mul, modified arithmetic.gen, cryobiology spanish.mul, evaluating information essentiality.mul, intent recognition.mul, understanding fables.mul, figure of speech detection.mul, empirical judgments.mul, simple ethical questions.mul, swahili english proverbs.mul, language identification.mul, phrase relatedness.mul, nonsense words grammar.mul, undo permutation.mul, object counting.gen, identify odd metaphor.mul, elementary math qa.mul, social iqa.mul, parsinlu qa.mul, metaphor understanding.mul, timedial.mul, causal judgment.mul, list functions.gen, implicatures.mul, date understanding.mul, codenames.gen, fact checker.mul, physics.mul, abstract narrative understanding.mul, emojis emotion prediction.mul, metaphor boolean.mul, strategyqa.gen,assic word recognition.gen, auto debugging.gen, cause and effect.mul, conlang translation.gen, cryptonite.gen, cs algorithms.mul, dyck languages.mul, gender inclusive sentences german.gen, hindi question answering.gen, international phonetic alphabet transliterate.gen, irony identification.mul, logical fallacy detection.mul, movie dialog same or different.mul, operators.gen, paragraph segmentation.gen, parsinlu reading comprehension.gen, repeat copy logic.gen, rephrase.gen, simple arithmetic json.gen, simple arithmetic multiple targets json.gen, sports understanding.mul, word unscrambling.gen, hyperbaton.mul, linguistic mappings.gen, anachronisms.mul, indic cause and effect.mul, question selection.mul, hinglish toxicity.mul, snarks.mul, vitaminc fact verification.mul, international phonetic alphabet nli.mul, logic grid puzzle.mul, natural instructions.gen, entailed polarity.mul, list functions.gen, conceptual combinations.mul, goal step wikihow.mul, logical deduction.mul, conlang translation.gen, strange stories.mul, odd one out.mul, mult data wrangling.gen, temporal sequences.mul, analytic entailment.mul, disambiguation qa.mul, sentence ambiguity.mul, swedish to german proverbs.mul, logical sequence.mul, chess state tracking.gen, reasoning about colored objects.mul, implicit relations.mul, riddle sense.mul, physical intuition.mul, simple arithmetic json multiple choice.mul, geometric shapes/gen, gem.gen, simp turing concept.gen, common morpheme.mul, qa.wikidata.gen, international phonetic alphabet transliterate.gen, similarities abstraction.gen, rephrase.gen, emoji movie.gen, qa.wikidata.gen, word sorting.gen, emoji movie.gen, qa.wikidata.gen, periodic elements/gen, hindi question answering/gen
Bellow, we plot the net percentage of tasks improved/deproved in each of the BIG-bench categories, out of the tasks that are changed by at least a threshold amount.
H Compute
We use one TPU v3-8 for all our pretraining runs. It takes approximately 23 hours for each pretraining run.
Our work primarily provides a scientific exploration and understanding of the properties of lexinvariant language models. More broadly, these properties could potentially help improve the robustness, generalizability, and reasoning ability of LMs in the future works. In general we don't foresee more specific negative societal impacts from this work other than general misuse of language models.
Figure 8: BIG-bench results with 0,1,2 and 3 shots.