What they do when in doubt: a study of inductive biases in seq2seq learners
Authors: Eugene Kharitonov
Rahma Chaabouni
Venue: ICLR 2021 Poster (2021)
Keywords: inductive biases
description length
sequence-to-sequence models

Abstract: Sequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize. We address that by investigating how popular seq2seq learners generalize in tasks that have high ambiguity in the training data. We use four new tasks  to study learners' preferences for memorization, arithmetic, hierarchical, and compositional reasoning. Further, we connect to Solomonoff's theory of induction and propose to use description length as a principled and sensitive measure of inductive biases. In our experimental study, we find that LSTM-based learners can learn to perform counting, addition, and multiplication by a constant from a single training example. Furthermore, Transformer and LSTM-based learners show a bias toward the hierarchical induction over the linear one, while CNN-based learners prefer the opposite. The latter also show a bias toward a compositional generalization over memorization. Finally, across all our experiments, description length proved to be a sensitive measure of inductive biases.

Published as a conference paper at ICLR 2021
WHAT THEY DO WHEN IN DOUBT: A STUDY OF INDUCTIVE BIASES IN SEQ2SEQ LEARNERS
Eugene Kharitonoy Facebook AI
Rahma Chaabouni Facebook AI ENS Ulm
ABSTRACT
Lequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize. We address that by investigating how popular seq2seq learners generalize in tasks that have high ambiguity in the training data. We use four new tasks to study learners' preferences for memorization, arithmetic, hierarchical. and compositional reasoning. Further, we connect to Salomonoff's s theory of induction and propose to use description length as a principled and sensitive measure of inductive biases.
In experimental study, find that LSTM- -asdd learners can learn perform counting, addition, and multiplication by a constant from a single training exam- ple. Furthermore, Transformer and ASTM-bbsse learners show bias toward the hierarchical induction over the linear one, while CNN-based learners prefer the opposite. The latter also show bias toward a compositional generalization over memorization. Finally, across all our experiments, description length proved to be a sensitive measure of inductive biases
INTRODUCTION
Seguuence-to-sqqenee (seq2seq) learners (Sutskever et al.) 2014) demonstrated remarkable perfor- mance in machine translation, story generation, and open-domain dialog (Sutskever et al. 2014 Fan al. 2018: Adiwardana et al. 2020). Yet, these models have been criticized for requiring tremendous amount of data and being unable to generalize systematically (Dupoux) 2018 8 Loula
al. 2018 Lake & Baroni 2017 Bastings et al.] 2018). contrast, humans rely on their inductive biases generalize from limited amount of data (Chomsky 1965; Lake al. 2019). Due the centrality of humans' biases in language learning, several works have studied inductive biases of seq2seq models and connected their poor generalization to the lack of the "right" biases (Lake &
Baroni 2017
Lake et al. 2019).
In this work, focus studying inductive biases of seq2seq models. We start from observation that, generally, multiple explanations can be consistent with limited training set, each leading to different predictions unseen data. A learner might prefer type of explanations over another in a systematic way, as a result of its inductive biases (Ritter et al. 2017, Feinman & Lake 2018).
To illustrate the setup work in, consider a quiz-like question: if f(3) maps to what does f(4) map to? The "training" example is consistent with the following answers: (f(x) = 6); 7 (f(x) = x + 3); (f(z) 2 2)); any number z, since we always can construct a function such that f(3) = 6 and f(4) = 2. By analyzing the learner's output on this new input, we can infer its biases.
This example demonstrates how biases of learners are studied through the lenses of the poverty of the stimulus principle (Chomsky 1965 1980): if nothing in the training data indicates that a learer should generalize
certain way, but does nonetheless, then this is due to the biases of the learner. Inspired by the work Zfang et al. (2019) in the image domain, we take this principle to the extreme and study biases of seq2seq learners in the regime of very few training examples, often as little as one. Under this setup, propose four new synthetic tasks that probe seq2seq learners' preferences to memorization-, arithmetic-, hierarchical- and compositional-bant "reasoning".
"Equal contribution.

---

Published as a conference paper at ICLR 2021
Next, we connect to the ideas of Solomonoff's s theory of induction (Solomonoff 1964) and Minimal Description Length [Rissanen] 1978) Grunwald (004) and propose to use description length, under a learner, as a principled measure of its inductive biases.
Our experimental study ssows that the standard seq2seq learners have strikingly different inductive biases. We find that LSTM- MrMaeeddd learners addition-based rules from as little as one example. CNN-bbaeed seq2seq learners would prefer linear over hierarchical generalizations, while LSTM-based ones and Transformers would do just the opposite. When investigating the compositional reasoning, description length proved to be a sensitive measure. Equipped with it, we found that CNN-, and, a lesser degree, LSTM- learners prefer compositional generalization over memorization when provided with enough composite examples. In turn, Transformers show strong bias toward memorization.
learn non-trivial counting-, multiplication-, and
SEARCHING FOR INDUCTIVE BIASES
To formalize the way we look for inductive biases of a learner M, we consider training dataset of input/output pairs, T = {I,, Vi Hi and hold-out set of inputs, H = {x;}==++ W.l.o.g, we assume that there are two candidate "rules" that explain the training data, but do not coincide on the hold-out data: C1 (2;)) = C2(2;) = Yi. S S n and Ei (Ix) # C2(2z), +1+11 i k.
To compare preferences of learner M toward those two rules, we the learner on the training data T and then compare its predictions on the hold-out data H to the outputs of the rules. We refer to this approach as "intuitive"". Usually, the measures of similarity between the outputs are task- specifici McCoy et al. (2020) used accuracy of the first term, Zhang et al. (2019) used correlation and MSE, and Lake & Baroni (017)) used accuracy calculated on the entire output sequence.
We too start with accuracy-based measure. We define the fraction of perfect agreement (FPA) between a learner M and a candidate generalization rule as the fraction of seeds that generalize perfectly in agreement with that rule the hold-out set H. Larger FPA of M is w.r.t. C, more biased M is toward C. However, FPA does not account for imperfect generalization nor allows direct comparison between two candidate rules when both are dominated by third candidate rule. Hence, below we propose a principled approach based on the description length.
Description Length and Inductive Biases At the core of the theory of induction (Solomonofi 1964) is the question of continuation of finite string that is very similar to our setup. Indeed, we can easily re-formulate our motivating example as a string continuation problem: "3 + 4 - The solution proposed by [Sooomonof] (1944) is to select the continuation that admits "the simplest explanation" of the entire string, i.e. that is produced by programs of the shortest length (description length).
Our intuition is that when a continuation is "simple" for a learner, then this learner is biased toward it. We consider a learner M to be biased toward C₁ over C2 if the training set and its extension according to C1 has shorter description length (for M) compared to that of Cz Denoting description length of a dataset D under the learner M as (DD)), we hypothesise that if LM (1CI(2))) L (LM((C((z)))
then M is biased toward C1.
Calculating Description Length To find the description length of data under a fixed learner, we use the online (prequential) code (Rissanen 1978 Grunwald, 2004 Blier & Ollivier, 2018).
The problem of calculating (D), D = {24, Vi 3 is considered as problem of transferring outputs y/2 one-by-one, in a compressed form, between two parties, Alice (sender) and Bob (receiver). Alice has the entire dataset {æ,, y: J. while Bob only has inputs {x,). Before the transmission starts, both parties agreed on the initialization of the model M, order of the inputs {=). random seeds, and the details of the learning procedure. Outputs {v:} are sequences of tokens from vocabulary V. W.l.o.g. we fix some order over {x). We assume that, given x, the learner M produces a probability distribution over the space of the outputs y, pm(y)2)
com/ Coode used in the experiments can be found http:///gggghht.ceooeeeeeeeeeeer
2

---

Published as a conference paper at ICLR 2021
The very first output V/1 can be sent by using not more than 1yal encoding After that, both Alice and Bob update their learners using the example (I1, 8/ available to both of them, and get identical instances of MM.
- llgg |V| nats, using naive
Further transfer is done iteratively under the invariant that both Alice and Bob start every step with exactly the same learners M and finish with identical Mr. At step Alice would use M to encode the next output Yt- This can be done using (- log P. (yr (z+)) nats (MacKay 2003). Since Bob has exactly the same model, he can decode the message to obtain yr and use the new pair ( ye) to update his model and get Mt. Alice also updates her model, and proceeds to sending the next y/+1 (if any), encoding it with the help of Mr. The cumulative number of nats transmitted is:
LM(D) = Σ
log +
The obtained code length of Eq. [ depends on the order in which y are transmitted and the procedure we use to update M. To account for that, we average out the data order by training with multiple random seeds. Further, for larger datasets, full re-training after adding a new example is impractical and, in such cases, examples can be transmitted in blocks.
we measure the description length of the training data T shuffled with the hold-out data H, both datasets would have symmetric roles. However, there is certain asymmetry in the extrapolation problem: we are looking for an extrapolation from T. not vice-versa. To break this symmetry, we always transmit outputs for the entire training data the first block.
While (D) is seemingly different from the "intuitive" measures introduced before, we can illustrate their connection follows. Consider case where first transmit the training outputs as the first block and all of the hold-out data outputs under C, C(H), as the second block. Then the description length equal cross-entropy of the trained learner on the hold-out data, recovering process akin to the "intuitive" measuring of inductive biases. With smaller blocks, the description length also catches whether a learner capable of finding regularity in the data fast, with few data ideas for measuring inductive biases (Chaabouni
points; hence it also encodes the al. 2019).
Finally, the description length has three more attractive properties when measuring inductive biases: (a) it is domain- indeeeiednd,,,,nt, i.e. can be applied for instance in the image domain, (b) it allows comparisons across models that account for model complexity, and (c) it enables direct comparison between two candidate rules (as we will show the Section
TASKS
We describe four tasks that we use study inductive biases of seq2seq learners. We select those tasks to cover different aspects of learners' behavior. Each task has a highly ambiguous training set which is consistent with infinite number of generalizations. We pre-select several candidate rules highlighting biases that are useful for language processing and known to exist in humans, or are otherwise reasonable. Our experiments show that these rules cover many cases of the learners' behavior.
The first two tasks study biases in arithmetic reasoning: Count-or Count-or-Mente Memorization quantifies learners' preference for counting vs. a simple memorization and Add-or-Multiae further probes the learners' preferences in arithmetic operations. We believe these tasks are interesting, as counting is needed in some NL.P problems like processing linearized parse trees (Weiss et al., 2018). The third task, Hierarchical-or-A
contrasts hierarchical and linear inductive reasoning. The hierarchical reasoning bias is believed to be fundamental in learning some syntactical rules in human acquisition of syntax (Chomsky 1965 1980). Finally, with the Composition-ors coroiionnoo--emmiiaiiaiiioo task, we investigate biases for systematic compositionality, which central for human capabilities in language. Figure] T illustrates these four tasks.
As we interested in comparing candidate generalization rules, the value of the additive constant c is not important, it learner- and -endddatt-indeeedeent. In experiments, we subtract from all measurements.
3

---

Published as a conference paper at ICLR 2021
train example
ana + MM
example aa + bb66
(mem
test example
Tmminti
example
b6b
Count-or Seenttor-Memmriann Memorisation
Add- or- MutMutipl
mud
train example
adbba
train example
thrice con
Neot ecample
Ohieran
best excempla
Ahrine
imer)
Jinsari
(comp))
Hieearchicol-or--inneee Composition or- Memorization
Figure 1: Illustration of the tasks. After training on the train examples (green blocks), learners tested held-out examples (red blocks). In pink blocks are generalizations according the candidate rules.
By at we denote sequence that contains token repeated k times. For training, represent sequences in a standard way: the tokens are one-hot-encoded separately, and we append a special seddof-sequence token to each sequence. Input and output vocabularies are disjoint.
Coumt-or-Memorization In this task, we contrast learners' preferences for counting vs. memoriza- tion. We train models to fit single training example with input a' and output t' (i.e., perform the mapping a' + 6') and test it on a with m E [ 10, ++ 10]. If learner learns the constant function, outputting b' independently of its inputs, then it follows the mem strategy. On the other hand, if generalizes to the a" + 6th mapping, then the learner is biased toward the count strategy.
Add-or-Multiply: This task is akin to the motivating example in Section ! The single training example represents a mapping of an input string a' to an output string 6²d, As test inputs, we generate nh for m in the interval [| 3, + 3). We consider the learned rule to be consistent with mul if for all m, the input/output pairs are consistent with a - 2mmm Similarly, if they are consistent with ath + bm- we say that the learner follows the addition rule, add. Finally, the learner can learn a constant mapping am - 12ll for any Again, we call this rule mem.
For fixed depth d, we train learners on four training examples xe yy -> {a, 6} Each training example has nested structure, where d defines its depth. A learner with a hierarchical bias (hierar), would output the middle symbol. We also consider the linear rule (linear) in which the learner outputs the (d+1)th symbol of its input.
Hierarchical-or- where
To probe learners' biases, we test them on inputs with different depths m E [d - 2, + 2]. Note that to examine the 1inear rule (i.e. if the learner outputs the (d+1) symbol of any test input of depth m), we need 2 5. Similar the previous tasks, there is vocabulary sharing between model's inputs and outputs (input and output tokens a and are different).
Composition-ors cnnppsttioono--Memriiiaaiiii Memorization:: We take inspiration from SCAN LLake & Baroni 2017 bench- mark used for studying systematic generalization of seq2seq learners The input vocabulary has N symbols a; that are one-to-one mapped into N output symbols b; (i.e., a; - b:). In addition, there is a modifier token (hrice: when thrice precedes an input symbol a; the corresponding output is repeated three times: thrice a; - b.b.b,.
We train a learner on all concoomposiiionll examples (a; + b.) and M (M < N) compositional examples (thrice a; + b,b,b;). At test time, we feed the learner with the remaining compositional examples (thrice a; i M). If the learer generalizes to the mapping thrice - bybyb, by for i > M, we consider it be biased toward a compositional reasoning (comp). As an alternative generalization,
"This mapping consist then on four combinations a "ba" - by a" aa + o; b ab" + and b' - b. 'In Appendix, report study of inductive hiases on the SCAN data.
4

---

Published as a conference paper at ICLR 2021
consider mapping where all inputs containing dg mapped into by: thrice a; by (t M) We call this generalization memorization (mem).
METHODOLOGY
4.1
SEQUENCE-TO-SEQUEBNE LEARNERS
We experiment with three standard seq2seq models: LSTM- based seq2seq (LSTM-s2s) (Sutskever
al. 2014), CNN-based seq2seq (CNN-s2s) (Gehring et al. 2017). and Transformer (Vaswani et al.
2017 share a similar Encoder/Deccodr architecture (Sutskever et al. 2014).
LSTM-s2s Both Encoder and Decoder are implemented as LSTM cells (Hochreiter & Schmidhuher
1997). Encoder encodes its inputs incrementally from left to right. We experiment with architectures
without (LSTM-s2s no att.) and with (LSTM-s2s att.) an attention mechanism (Bahdanau et al.
2014]. For the first three tasks, both Encoder and Decoder are single-layer LSTM cells with hidden
size of 512 and embedding of dimension 16.
CNN-s2s Encoder and Decoder are convolutional networks (LeCun et al. 1990), followed by GLU non-linearities (Dauphin et al. 2017) and an attention layer. To represent positions of input tokens, CNN-s2s uses learned positional embeddings. Encoder and Decoder networks have one layer with 512 filters and kernel width of 3. We set the embedding size to 16.
Transformer Encoder and Decoder are implemented
sequence of sself-Jattention and feed- forward layers. We use sinusoidal position embeddings. Both Encoder and Decoder contain one transformer layer. The attention modules have 8 heads, feed-forward layers have dimension of 512 and the embedding is of dimension 16.
In Appendix we report experiments where we vary hyperparameters of the learners.
4.2 TRAINING AND EVALUATION
For all tasks, we follow the same training procedure. We train with Adam optimizer (Kingma & Ba 2014) for 3000 epochs. The learning rate starts at 10 and increases for the first 1000 warm-up updates till reaching 10-3, We include all available examples in single batch. We use teacher forcing (Goodfellow et al., 2016) and set the dropout probability to 0.5. For each learner, we perform training and evaluation 100 times, changing random seeds. At generation, to calculate FPA, we select the next token greedily. We use the model implementations from fairseq (Oit et al. 2019).
As discussed in Section when calculating L, we use the training examples as the first transmitted block at t 1. In Count-or-Mente Memorization and add-or-Multiply this block contains one example, in Eiterarchicallor-Lnneee examples. Next, we transmit examples obtained from the candidate rules in a randomized order, by blocks of size 1, 1, 1, and 4 for Count-or- Count-or-Memorizattee Memorization, Add-or-Multiply. Composition-or- Memorization, and Hierarchical-or sierarchicallor-Lineed respectively. At each step, the learner is re-trained from the same initialization, using the procedure and parammerssetters discussed above.
it has 4 examples, and in Composition-or- -o--ooo-ooo-emmniaaaiiinooo it has N + M
Our training setup is typical for seq2seq training. It does not include any additional pressure towards description length minimization. Description length is solely used measure of inductive biases.
5 EXPERIMENTS
Count-or-Memorizztion We investigate here learners' biases toward count and mem rules. We provide a single example a' - b/ the training set, varying I (10, 20, 30, 40). We report the learners' performances in Table la We observe that, independently of the length of the training example I, CNN-s2s and Transformer learners inferred perfeetly the mem rule with FPA-mem > 0.90 (i.e. more than 90% of the random seeds output b' for any given input a
However, LSTM- based learners demonstrate a more complex behavior. With / = 10, both learners (with and without attention) exhibit preference for mem. Indeed, while these learners rarely generalize perfectly to any of the hypothesis (0.0 FPA (no att.), 0.2/0.0 FPA for men/count (att.)), they have significantly lower I-mem. As / increases, LSTM-based learners become more biased
5

---

Published as a conference paper at ICLR 2021
FPA
L, nais 4
length
count
mem
count
mem
LSTM-s2s
40
1.00
0.00
0.01
97.51
30
0.97
0.00
0.01
72.67
20
0.07
0.00
2.49
55.67
10
0,00
0.00
88.27
48.67'
LSTM-s2s att.
40
0.99
0.00
7.84"
121.48
30
0.96
0.02
1.14"
83,48
20
0.70
U.16
5,73'
49.33
10
0.00
0.20
98.12
8.46
CNN-s2s
[10, (1, 20, 40)
0.00
0.90
>> 592.92
<1.31'
Transformer
[10, 20, 30, 40)
0.00
0.97
>> 113.30
<11.14
(a) Count-or- Memorizztionnn
FPA
L, nats
length
add
mul
mem
add
mul
mem
LSTM-s2s no att.
20
0.00
0.94
0.00
25.42
0.31
57.32
15
0.07
0.65
0.00
19.24
4.67
43.65
10
0.95
0.01
0.00
0.68
26.58
25.15
0.04
0.00
0.00
17.12
50.83
18.60
LSTM-s2s att.
20
0.00
0.98
0.00
30.26
1.40"
58.84
15
0.15
0.83
0.00
20.18
4.07
46.36
10
0.40
0.28
0.18
13.69
18.16
26.44
0.00
0.00
0.97
45.88
77.86
0.01
CNN-s2s
{5. 10, 15, 20}
0.00
0.00
1.0
318.12
346.19 19
0.00
Transformer
{5. 10, 10)
0.00
0.00
1.0
38.77
50.64
<3,50
(b) Add-or-Multilly
FPA
L. nats 1
hierar
linear
hierar
linear
LSTM-s2s att.
0.05
0.00
31.04
61.84
LSTM-s25
0.30
0.00
26.32'
57.2
CNN- s2s
0.00
1.00
202.66
0.00*
Transformer
0.69
0.00
4.84"
(c) Hicrarchical-or-I Lincar with depth d 4
FPA ?
L, nats |
M, examples
comp
more
mem
LSTM-s2s no att.
36
0.00
0.00
42.65
38.55
0.00
0.00
238.5 238.54
89,36'
0.00
0.00
157.55*
LSTM-s2s att.
0.00
0.00
62.34
70.92
24
0.00
0.00
263.33
157.82"
6
0.00
0.00
659.85
164.43
CNN-22s
36
0.75
0.00
1.44"
49.92
0.13
0.00
13.75*
84.55
0.00
0.00
131.63
29.66'
Transformer
0.00
0.82
147.83
6.36'
24
0.00
0.35
586. 22
26.46
0.00
0.00
1235.07 01
53,91
(d) Composiiton-o--Mmmrriiiii
Table 1: FPA measures the fraction of seeds that generalize according to a particular rule. Description length L is averaged across examples and seeds. The lowest L are in bold and denotes stat. sig. difference in L (p < 10-3, paired t-test).

---

Published as a conference paper at ICLR 2021
toward count. Surprisingly, for 2 30, most learner instances show sufficiently strong inductive biases to infer perfectly the non-trivial count hypothesis. With = 40, 99% of random seeds of LSTM-s2s att. and all (100%) of LSTM-s2s no att. seeds generalized perfectly to count.
Further, we see that if L shows similar trends, it has a higher sensitivity. For example, while both LSTM-based learners have similar FPA with 7 40, L demonstrates that LSTM-s2s att. has stronger count bias.
Add-or-Multiply In this task, we examine learners' generalization after training the single example a' - l0 We vary / E {5, 10, 15, 20). In Table 1b] we report FPA and L for the three generalization hypotheses, add, mu and mem. We observe, similarly to the previous task, that CNN-s2s and Transformer learners always converge perfectly to memorization.
In contrast, LSTM-based learners show non-trivial generalizations. Examining first LSTM-s2s att., when (=5, we note that mem has a high FPA and an L considerably lower than others. This is consistent with the learner's behavior in the Count-or-Memorriane interesting behavior emerges. First, L-mem decreases as I increases. Second, mul-type preference increases with I. Finally, L-add presents U-shaped function of 1. That is, for the medium example length 1, the majority of learners switch to approximating the add rule (for / = 10). However, when I grows further, a considerable fraction of these learners start follow mul 1 -type rule. Strikingly, 98% of LSTM-s2s att. seeds generalized perfectly to the non-trivia mul rule. As for LSTM-s2s no att., we do not observe a strong bias to infer any of the rules when /=5. However, when increasing 1, the LSTM-s2s no att. behaves similarly LSTM-s2s att.: at first it has a preference for add (FPA-add=0.95, for 1=10) then for mul (e.g. FPA-mu1=0.94, for /=20).
task. As we increase 1, more
Hierarchical-or-Linne We look now at leamers' preference for either hierar linear gener- alizations. The architectures we use were only able to consistently learn the training examples with the depth d not higher than 4. Hence, in this experiment, we set d to 4.
We report in Table Tc the learners' FPA and L. We observe that CNN-s2s exhibits a strikingly different bias compared to all other learners with perfect agreement with the linear rule. In contrast, Transformer learners show clear preference for hierar with a high FPA (0.69) and a low L (1.21). Surprisingly, this preference increases with the embedding size and Transformers with embedding size 2 64 admit an FPA-hierax of 1.00 (see Appendix for more details). LSTM-s2s att. learners demonstrate also a similar preference for hierar with an FPA of 0.30 and considerably lower I, than
Finally, while only 5% of LSTM-s2s no att. instances generalized to perfect h wierar r (and none to niner).. r), I confirms their preference for the hi hierar r hypothesis.
In this task, we set the number of primitives N = 40 and vary the number of compositional examples M E {6, 24, 36} seen during training. Results are reported in Table 1d First, we observe that FPA is only informative for CNN-s2s when trained with a large M. Indeed, for M = 6, CNN-s2s does not infer any of the candidate rules. However, according to description length L. we note significant preference for mem over comp. More compositional examples CNN-based learners see at training, more biased they become toward comp. The remaining learners have zero FPA for both candidate rules. However, according to description length, L.STM- based learners have preferences similar to CNN-s2s, although weaker. That is, they show a preference for mem for low M, that declines in favor of comp as M increases. In contrast, Transformers show strong bias for mem with all tested M.
Compestion-oor-Meoriiiz
Overall, across all the above experiments, we see that seq2seq learners demonstrate strikingly different biases. In many cases, these biases lead non- non-trivial generalizations when facing ambiguity in the training data. This spans tasks that probe for memorization, arithmetic, hicrarchical, and compositional reasoning. We found that a single example is sufficient for LSTM-based learners to learn counting, addition, and multiplication. Moreover, within the same task, they can switch from one explanation to another, depending on the training example length, with Addditionoor-Mulliliiaaton being the task where this switch happens twice. In contrast, CNN-s2s and Transformers show a strong bias toward memorization. Furthermore, all learners except for CNN-s2s demonstrate a strong bias toward the hierarchical behavior. In the task of compositional generalization. CNN-s2s shows strong bias toward compositional reasoning that appears after a few compositional training examples. On the other hand, Transformers show preference for memorization over compositional generalization.
7

---

Published as a conference paper at ICLR 2021
We see that the conclusions derived from comparing the description length of the candidate rules in agreement with the results under accuracy-basee metrics, but provide a more nuanced picture.
We observe that learners' biases depend, in many cases, the length/number of the input examples. In Appendix, we examine impact of other hyper-parameters. In particular, we study impact of (1) learners' architecture size, by varying the number of layers, hidden and embedding sizes, and (2) dropout probability. Our results show that in some cases a learner's architecture size can influence the strength of inductive biases, but rarely modify them: among the 136 tested settings, we observe only 3 cases of switching the preferences. We also found, in line with (Arpit al., 2017). that large dropout probabilities can prevent mem-type generalization. Finally, in Appendix we show that variant of Transformer learners, namely the joint source-target self-attention learner (He et al.) 2018: Fonollosa al. 2019), displays the same preferences as the standard Transformer learners. This variant resembles the "decoder-only" architecture used in language modeling (Radford et al., 2019 [Brown et al. 2020). This result demonstrates that our tasks and bias measures could be applied for studying inductive biases of language model architectures.
Robustness to hyper-parameterr
6 RELATED WORK
Dessi & Baroni (2019) found that, unlike LSTM-s2s learners, CNN-s2s can perform compositional generalization on SCAN. Our experiments indicate that this only happens when enough compositional examples are provided in the training. Moreover, such a case, attention-enabled LSTM-s2s also start to prefer compositional generalization over memorization.
McCoy et al. (2020) studied inductive biases of recurrent neural architectures two synthetic tasks, English question formation and tense inflection. They found that only tree-based architectures show robust preference for hierarchical reasoning. in contrast to LSTM-s2s learners that generalized linearly. Our experiments the hyperparameter robustness, reported in Appendix, indicate that the preferences over linear/hierarchical reasoning are strongly affected by the dropout probability, with learners shifting to linear behavior with low probabilities. As McCoo et al. (2020) experimented with a low dropout probability of 0.1, we believe this explains the misalignment of the conclusions.
Overall, our study shows that inductive biases are more complicated than it seemed in these prior works and a more careful analysis is crucial. We believe that our extremely controlled setup with very little confounds is good addition to those studies.
Another line of research investigates theoretically learners' capabilities, that is, the classes of the hypothesis that a learner can discover (Siegelmann & Sontag 1992. Suzgun al., 2019 Merrill et al. 2020). For example. Weiss et al. (2018) demonstrated that LSTM cells can count. In turn, we demonstrate that LSTM-s2s learners are not only capable but also biased toward arithmetic behavior.
7 DISCUSSION AND CONCLUSION
In this work, we studied inductive biases of standard seq2seq learners, Transformer- LSTM- and CNN-based. To do so, we introduced four new tasks, which allowed to cover an interesting spec- trum of behaviors useful for language learning. In particular, we considered arithmetic, hierarchical, and compositional "reasoning". Next, we connected the problem of finding and measuring inductive biases to Solomonoff's 's theory of induction and proposed to use dataset's 's description length under learner as tool for sensitive measurement of inductive biases.
In our experiments, we found that the seq2seq learners have strikingly different inductive biases and some of them generalize non-trivially when facing ambiguity. For instance, a single training example is sufficient for based learners to learn perfectly how to count, to add and to multiply by a constant, Transformers and, to lesser degree, LSTM-s2s demonstrated preferences for the hierarchical bias, bias that has been argued to govern children's acquisition of syntax. Interestingly, such biases arose with no explicit wiring for them. Our results support then Elman et al. (1998)ss s theory which states that human's 's inductive biases arise from low-level architectural constraints in the brain with no need for an explicit encoding of a linguistic structure. However, how the brain, or, more generally, learner is wired admit specific inductive bias is still important open question.
8

---

Published as a conference paper at ICLR 2021
Across our experiments, also observed that description length is consistent with "intuitive" measurements of inductive biases, and, at the same time, it turned out to be more sensitive. This also indicates that, in the presence of ambiguity in the training data, learner is more likely to follow the alternative with the shorter description length (i.e. the simplest one) when applied on unseen data, showing consistency with the prescriptions of the theory of induction (Solomonoff) (964)). A similar simplicity preference is argued to play role in human language acquisition (Perfors et al. 2011).
Our work provides simple tools to investigate learners' biases. We first show that FPA is intuitive measure to study biases when provided with simple tasks. Second, we present description length as robust measure to fairly compare learners' biases. This metric considers learners' size and their case of learning as opposed to accuracy-based metrics. Besides, it is model- and task-agnostic measure that succeeds unveiling learners' biases even when presented with more complex tasks with spurious correlations.
Our findings can guide for architecture selection in the low-data regimes where inductive biases might have a higher influence model's generalization performance. Large sparse datasets can also benefit from predictable behavior few-shot scenarios akin to what we consider.
Finally, our results demonstrate that relatively large deep learning models can generalize non-trivially from as little as one example as long as the task is aligned with the their inductive biases. We believe this should reinforce interest in future work on injecting useful inductive biases in our learners and, we hope, our findings and setup can provide a fertile ground for such work.
ACKNOWLEDGEMENT
The authors are grateful to Marco Baroni, Emmanuel Dupoux, Emmanuel Chemla and participants of EViL seminar for their feedback on our work.
REFERENCES
Daniel Adiwardana. Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards human- like open- peen-omainn chatbot. arXiv preprint WVXiv:2001.0997,, 2020.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look memorization in deep networks. In International Conference Machine Learning, 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1.KKv:1409.04777 2014.
Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: SCAN both left and right. In EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, 2018
Léonard Blier and Yann Ollivier. The description length of deep learning models. In NeurlPS. 2018.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-111 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are learner arXiv preprint 2xXv:20005.14163 2020.
Rahma Chaabouni, Eugene Kharitonov, Alessandro I Lazaric, Emmanuel Dupoux, and Marco Baroni. Word-order biases deep-agent emergent communication. arXiv preprint arXiv: 1905. /2330, 2019.
Noam Chomsky, Aspects of the Theory of Syntax, volume 11. MIT press, 1965.
Noam Chomsky, Rules and representations: behavioral and brain sciences, 1980.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In JCML. JMLR. org, 2017.

---

Published as a conference paper at ICLR 2021
Roberto Dessi and Marco Baroni. Cans found to jump around skillfully than Compositional generalization in seq2seq convolutional networks. ACL, 2019.
Emmanuel Dupoux. Cognitive science in the of artificial intelligence: A roadmap for reverse-engineering the infant language-learer. Cognition, 173:43-59, 2018.
Jeffrey L Elman, Elizabeth A Bates, Mark H Johnson, Annette Karmiloff- Kim Plunkett, and Domenico Parisi. Rethinking innateness: A connectionist perspective development, volume 10. MIT press, 1998.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, 2018.
Reuben Feinman and Brenden Lake. Learning inductive biases with simple neural networks. arXiv preprint arXiv: 1802.02745, 2018.
José AR Fonollosa, Noe Casas, and Marta Costa-jussà. Joint source-target self attention with locality constraints. arXiv preprint arXiv:1905.06653 06596, 2019.
Jonas Gchring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. In ICML, 2017.
Ian Goodfellow, Yoshua Bengio. and Aaron Courville. Deep learning. MIT press, 2016.
Peter Grunwald. tutorial introduction to the minimum description length principle, arXiv preprint math/0406077, 2004.
Tianyu He,
Tan, Yingce Xia, Di He. Tao Qin, Zhibo Chen, and Tin-Yan Liu. Layer-wise coordination between encoder and decoder for neural machine translation. In Advances in Neural Information Processing Systems, pp.
2018
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 173:17355 1997.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalisc? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142.6914 2014.
Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. arXiv preprint arXiv:1711.00:500 2017.
Brenden M Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional instructions. arXiv preprint arXiv:1901.04593 2019.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubhard, and Lawrence D Jackel. Handwritten digit recognition with hroppproonn network. In NeurIPS, 1990.
Joao Loula, Marco Baroni, and Brenden M Lake. Rearranging the familiar: Testing compositional generalization in recurrent networks. arXiv preprint 1.Xiv:8807.05544 2018
David JC MacKay, Information theory, inference and learning algorithmms. Cambridge university press, 2003.
Thomas McCoy, Robert Frank. and Tal Linzen. Does syntax need to grow trees? sources of hierarchical inductive bias seuuence-to-sqqueee networks. arXiv preprint arXiv:2001.06322. 2020.
William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Eran Yahav. A formal hierarchy of rnn architectures, 2020.
Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangior, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT 2019: Demonstrations, 2019.
Amy Perfors, Joshua Tenenbaum, and Terry Regier. The learnability of abstract syntactic principles. Cognition, 118(3):306-335 3388 2011.
Alcc Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188)9 2019.
Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465-471, 1978.
Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In /CMI, 2017.
10

---

Published as a conference paper at ICLR 2021
Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. In COLT, 1992.
Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7(1):1-22, 1964.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In NeurIPS, 2014.
Mirac Suzgun, Schastian Gehrmann, Yonatan Belinkov, and Stuart M Shieher. Lstt networks can perform dynamic counting. arXiv preprini arX1v:1900003648, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Gail Weiss, Yoav Goldberg, and Eran Yahav, On the practical computational power of finite precision rmns for language recognition. arXiv preprint urXiv:/805.04900 2018.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in neural information processing systems, pp. 4148-4158. 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterrizatio.. arXiv preprint arXiv: 1902.0468, 2019.

---

Published as a conference paper at ICLR 2021
FPA
nais
length /
mull
mul2
mul3 3
mem
mu mull
mul2
mul3
mem
LSTM-s2s no alt.
20
0.00
0.01
0.78
0.00
52.27
22.20
1.17
77.75
15
0.00
0.13
0.45
0.00
40.46
13.22
6.14"
66.10
10
0.00
0.92
0.00
0.00
26.48
0.65
22.26
53.81
0.49
0.00
0.00
0.00
1.97
26.50
54.97
36.13
LSTM-s2s att.
20d
0.00
0.19
0.62
0.00
36.76
20.35
9.35*
49.34
15
0.00
0.14
0.76
0.00
37.84
18,42
5,53*
56.43
10
0.02
0.45
0.49
0.00
29.83
11.67
8.96
45.47
5
0.01
0.03
0.00
0.64
32.97
48.26
60.38
2.44"
CNN-s2s
20
0.00
0.00
0.00
0.99
263.82
262.01
261.71
0.02
15
0.00
0.00
0.00
1.00
250.97
253.32
253.08
0.00
10
0.00
0.00
0.00
1.00
243.17
245.16
248.28
0.00
5
0.00
0.00
0.00
1.00
258.10
257.79
264.06
0.00
Transformer
20
0.00
0.00
0.00
0.97
37.90
51.47
57.57
5.31
15 0.00
0.00
0.00
1.00
40.36
51.62
57.42
2.50
10
0.00
0.00
0.00
1.00
38.05
49.88
55.61
2.47
5
0.00
0.00
0.00
1.00
37.96
51.83
60.19
0.74
Table 2: Multiplication by 3. FPA measures the fraction of seeds that generalize according particular rule. Description length I. is averaged across examples and seeds. The lowest L, are in bold and denotes stat. sig. difference in L (p 10-3, paired t-test).
A CAN SEQ2SEQ LEARNERS MULTIPLY BY 3?
In our experiments on the Multiply-o--Ad task we saw that LSTM-s2s learners are able to learn to multiply by 2 from a single example. A natural further question is whether these learners can Iearn to multiply by larger numbers? Here only provide a preliminary study in hope to inspire more focused studies.
We build task that similar Multiply-or-Add, but centered around multiplication by instead of 2. The single training example represents a mapping
input string a' to an output string bd,, As test inputs, we use a" with m coming from an interval [ 3,4 + 3].
Since 3x can be represented as a combination of addition and multiplication in several ways (3r = x: + 2r 2x z)), consider 1 different candidate rules. As before, by mem we denote the constant mapping from T 5. mull represents the mapping a" 2 mul2 corresponds to an -> p and mu denotes am - 53s The "explanation" muil akin to the add rule in the Mulriply-or-Adi task. use the same hyperparameters and training procedure described in Section 4.
We report the results in Table 2 I Like the results observed in the Multiply-or-Add task, both CNN-s2s and Transformer learners show strong preference for the mem rule while I.STM-hased learners switch their generalization according the length of the training example Indeed, for CNN-s2s and Transformer, we note an FPA-mem200.97 independently of (with L-mem significantly lower than others). LSTM-s2s att. learners start inferring the mem rule for = 5 (FPA=0.64. L=2.44), then switch to comparable preference for mu.12 and mu13 when / = 10. and fimally show significant bias toward the mul3 hypothesis for E (15,20) (e.g. FPA-mu13=0.76 76 for 15). LSTM-s2s
learners also subject similar switch of preference. That for these learners have sigmificant bias toward mul1 (FPA=0.49). Strikingly, when / 92% LSTM-s2s no att. leamners inferred perfectly the mul2 rule after one training example. Finally, we observe again another switch to approximate mul3 for E 213,203.
Overall, while CNN-s2s and Transformer learners show significant and robust bias toward mem, LSTM-based learners generalize differently depending the training input length. In particular, results suggest that these learners avoid adding very large integers by switching to multiplicative explanation in those cases. Answering our initial question, we see that LSTM-baser learners can learn to multiply by 3 from single example.
*Only % of learners succeeded in learning the training example in this setting. "Only 51% of learners succeeded in learning the training example in this setting.
12

---

Published as a conference paper at ICLR 2021
B ROBUSTNESS TO CHANGES IN ARCHITECTURE
In this section, we examine how changing different aper-parameters affects learners' preferences for mem- orization, arithmetic, hierarchical and compositional reasoning. In particular, we vary the number of layers. hidden and embedding sizes of the different learners and test their generalization Count-or-Memoriian Add-or-Multiply, Hievarchical-or--ineee
and compositionorr-Mmorrtttnn tasks.
and Add-or-Multiply respectively, all the experiments, we fix = 40 and I 20 for caappsitionoor-Mmmrrziiiiio Finally, we keep the same d = 4 for fiterachiicllor-Lineeee and . = 36 for Composition-or training and evaluation procedure detailed in Section 4.2 of the main text. However, we use 20 different random seeds instead of 100.
B.1 NUMBER OF HIDDEN LAYERS (Nooyer)
We experiment with the standard seq2seq learners described in the main paper and vary the number of layers Lager 31,3, 10). Results are reported in Table
First, when looking the interplay between mem and count (Table Bap. we observe that, independently of N Mager, more than 97% of CNN-s2s and Transformer learners inferred perfectly the mem rule (i.e. output 640 for any given input a"") Further, if the preference for count decreases with the increase of NLager. LSTM-s2s alt. learners display in all cases significant bias toward ccunt with large FPA and L significantly lower than L-mem However, we note decline of the bias toward count when considering LSTM-s2s no att. learners. For N Layer = 1, 100% of the seeds generalize to perfect count, versus 6% for NLager = 10. Note that this lower preference for count is followed by an increase of preference for mem. However, there is no significant switch of preferences according to L.
Second, we consider the Add-or ortl-rrplyyyy task where examine the three generalization hypothesis add, mul mem. Results are reported Table Similarly to the previous task, Transformer and CNN-s2s learners are robust to the number of layers change. They perfectly follow the mem rule with FPA-nem=1 1.00. However, LSsed learners show a more complex hehavior: If single-layer LSTMMs2s att. and no att. demonstrate a considerable bias for mul (FPA-mu 1TPA-mu1>0.94 00994), this bias fades in favor of add and memo for larger architectures.
Third, in Table Be] observe that larger learners slightly less biased toward hierar and linear. However, we do not observe any switch of preferences. That is, across different Ntagera, CNN-s2s learners prefer the 1 linear rule, whereas Transformers and, to lesser degree, LSTM-based learners show a significant preference for the hierar rule.
Finally, we do not observe an impact of NLayer on the Composiioonoor-Memrttaaoio task (see Table 3d demonstrating the robustness of biases w.r.t. learners' size.
In sum, we little impact of NLayer on learners' generalizations. Indeed, LSTM--aased learners can learn to perform counring from a single training example, even when experimenting with 10-layer architectures. For most tested N Layer, they favor mu and add over mem. In contrast, Transformer and CNN-s2s perform systematic memorization of the single training example. Furthermore, independently of Nt.yer, Transformer and LSTM-based learners show a bias toward the hierar hypothesis the 1 linear one, while CNN-based learners the opposite. Finally, compositional experiments further support how inductive biases are barely influenced by the change of the number of layers.
B.2 HIDDEN SIZE (SHidden)
We experimented in the main text with standard seq2seq learners when hidden size Syudden = 512. In this section, we look at the effect of Sæidden varying it in {128, 512, 1024/. We report learners performances in Table
First, Table demonstrates how minor effect hidden size has learners counting/memorizztionn performances. Indeed, for any given Stridden, between 84% and 100% LSTM-based learners learn perfect counting after only one training example. Similarly, and with even a lower variation, more than 91% of Transformer and CNN-s0 s2s learners memorize the single training example outputting 500 for any given a.
The same observation can be made when studying the interplay between the hierar and linear biases (Table 4c) and comp and mem biases (Table[4d) Concretely, Tables Add show that learners' generalizations stable across Stidee values with the exception LSSMM-based learners that loose significance for some tested S SHidden values. Yet. there is no switch of preference.
Finally, demonstrated in Table 4b all Transformer and CNN-s2s learners perform perfect memorization when tested on the Add-or-Multiply task independently of their hidden sizes. Both L.ST--hased learners are significantly biased toward ma1 for Smidden {512, 1024). However, when experimenting with smaller
13

---

Published as a conference paper at ICLR 2021
FPA
I, nals
Naayr
LSTM-22s no att.
1.00
0.00
0.01'
97.51
3
0.80
0,00
0.00"
60.80
10
0.06
0.47
16.89
22.02
LSTM-x2s
0.99
0.00
7.84
121 21.48
3
0.50
0.00
11.30
57.57
10
0.39
0.22
22.86
45.13
CNN:22
0.00
0.98
660.73
0.02"
3
0.00
1,00
1685,18
0.00
10
Transformer
0.00
0.97
116.34
11.10"
3
0.00
1.00
139.58
0.31
(a) Saant-or-Memorizationn
FPA
nats
Naage
LSTM-22s
0.00
0.94
0.00
25.42
0.31
57.32
3
0.15
0.20
11.00
9.82
10.11
33.16
10
0.28
0.00
5.83
23.16
10.01
LSTM-s2s att.
0.00
0.98
30.26
1.40'
58.84
0.65
0.20
0.00
5.88
15.67
27.82
10
0.11
0.11
0.28
9.66
27.72
8.26
CNN-s2s
1
0.00
0.00
1.00
318.12
346,19
0,00"
0.00
0.00
1.00
1058.27
824.93
2.31
10
Transformer
0.00
0.00
1.00
38.77
50.64
3.50
3 0.00
0.00
1.00
40.03
57.73
0.18
10
(b) Add Add-or-Multipe
FPA
Naager
hierar
linear
hierar
linear
LSTM-s2s no att.
0.05
31.04
61.84
0.00
0.00
29.08'
50.92
10
LSTM-s2s att.
0.30
0.00
26.32'
57.20
0.00
32.32'
50.12
10
CNN-s2s
0.00
1.00
202.64
0.00
222.32
10
Transformer
0.69
0.00
4.84
35.04
0.56
5.48
29.16
10
(c) Hierarchical-or-
FPA
L, nats
Nayer
comp
comp
LSTM-s2s
0.00
0.00
42.65
38.55
0.00
0.00
37.01
34.16
LSTM-s2s att.
0.00
0.00
62.34*
70.92
0.00
0.00
46.10
58.79
CNN-22s
0.75
0.00
1.44
49.92
0.95
0.00
1.79
66.44
Transformer
0.00
0.00
147.83
6,36
(d) Composiiion--r-Meennee Memorization
Table 3: Effect of the number of layers (Ntayer)): FPA measures the fraction of seeds that generalize according to particular rule. Description length L is averaged across examples and seeds. The lowest L are in bold and denotes stat. sig. difference in L (p < 10 paired t-test.). '_" denotes settings where learners have lower than 70% success rate on the train set.
14

---

Published as a conference paper at ICLR 2021
Stidden (=128), we detect switch of preference for LSTM-s2s no att. learners. The latter start approximating add-type rule (with significantly lower L). Lastly, we do not distinguish any significant difference between add and mul for L2s att. when Syidden = 128.
Taken together, learners' biases are quite robust to Siidden variations. We however note a switch of preference from mul to add for LSTM-s2s no att. learners when decreasing Spriddnn Furthermore, we see a loss of significant preference in five distinct settings.
B.3 EMBEDDING SIZE (Sgmb)
We look here at the effect of the embedding size, Sgmb, on learners' generalizations. In particular, we vary S Fmb E {16, 64, 256]. Results are reported in Table
Across sub-tables 5a and 5c
we see small influence of F on learners' biases. For example. we consider task when varying SEmb (see Table between 95% and 100% of LSTM-s2s no
the Count-or-Memoricnne att. Iearners inferred perfectly the count hypothesis. More striking, between 99% and 100% of LSTM att. Iearners Iearned the count rule after onc training example. The same trend is observed for the remaining learners and across the other tasks; Add-or-Multiply (Table5b) and Hierarchicallor-Linne still discern in some cases, systematic, but low, effects Emb. First, the larger Smmb is, the lower FPA-mu of LSTM-s2s no att. learners is (from 0.94 for SEmb 16 to 0.84 for SEmb = 256). However, LSTM-s2s no att. learners still have considerable preference for mul for any tested SEmb. Second, we of Transformer's preference for hierar with the increase of Sgmb. Surprisingly, for Sgmd 64, 100% of Transformer learners generalize to perfect bierar hypothesis.
(Table 5c). Yet, we
increase
Finally, when considering the cenpoosiion-or--mmrriattt tendency to prefer comp generalization when SEmb is large. For example, if, for L.STM-baeed learners, we note 0.00 comp-FPA when SEmb 16, comp-PPA 0.85 with SEms = 256. Interestingly, Transformers switch their preference, showing significant bias toward compositional reasoning with F 256.
task, we observe Table[5d] for all learners, a
this section, we studied the impact of the number layers, hidden and emhedding sizes on learners' generalizations. We found that, if these hyper-parameters can influence, in some cases, the degree of one learner's 's preference w.r.t given rule. inductive biases are quite robust to their changes. In particular, among all tested combinations, we observe only cases of preference switch (out of 136).
C ROBUSTNESS TO CHANGES IN TRAINING PARAMETERS
We examine here the effect of the training parameters on learner'' biases As previously, we only consider the Coumt-or-Memorzzatio task with I = 40, the Add-or-Mulriply task with 1 20, the Hierarchical-or--ineee task with = and the compoxttion-or-Meoriizan detailed in the main text; however, we use 20 different random seeds instead of 100.
task with M 36. We experiment with the architectures
We consider in this section two different hyperparameters: (1) the choice of the optimizer, and (2) the dropout probability.
Optimizer We experiment with replacing the Adam optimizer [Kingma & Ba 2014) with SGD (Hottou al. 2018). We found experimentally that learners failed to learn the training examples consistently in most of the settings. Yet, when successful, they showed the same preferences. In particular, Transformer and CNN-s2s were the only learners that had good performances on Count-or- Count-or-Memorizen Memorization and Add-or-Mulliph train sets (success rate higher than 70%). These learners showed prefect generalization to the mem. rule in both tasks. Prior works had shown that SGD lcads to better generalization compared other adaptive variants (Wilson et al. 2017]. However, based on our few successful instances with SGD, we do not observe a difference between SGD and Adam results. One main difference compared to the prior work is the definition of generalization. For instance, in the Dount-or-Memoriznn it explains the training set fully. Hence, preferring memorization when trained with SGD, opposed counting, does not contradict the advantage of SGD when testing generalization performances.
task, memorizing the training example could be seen as perfect generalization
Dropout We then examine how the dropout probability affects learner'' preferences. We use, as mentioned in the main paper, Adam optimizer and vary the dropout probability dropout E {0.0, 0.2, 0.5}. Results are reported in Table
Both Count-or-Memorizatie (Table 6a) and Add-or-Mulriply (Table 6b tasks show the same trend. First, Transformer and CNN-s2s learners prefer consistently the mem rule. Second, when looking at LSTM-base based learners, we distinguish a more complex behavior. For dropout 2 0.2, based learners show a significant preference for arithmetic reasoning (count for the Count-or- orMeoorrzaaiinnio task and mul for the Add-or- Multiply task). However, when dropout (l.0, see different preferences. In particular, both M.STM- learners show preference for mem (not significant for LTM-ss2s att.), LSTS-s2s learners are significantly
15

---

Published as a conference paper at ICLR 2021
FPA
I., nals
Smaade
count
LSTM-12s no alL
128
0.84
0.00
8.25
109.84
512
1,00
0.00
0.01
97.51
1024
1,00
0.00
0.00
149.86
LSTM-s2s
128
0.90
0.00
0.01
89.31
512
0.99
0.00
7.84
121.48
1024
1,00
0.00
0.00
30093
CNN-<2s
128
0.00
1.00
805.01
0.00
512
0.00
0.98
660.73
0.02"
1024
0.00
1,00
993.85
0.00
Transformer
128
0.00
0.91
110.68
9.57
0.00
0.97
116.34
11.10"
1024
0.00
0.94
122.38
1.42*
(a) Count-or-Memorizalle Memorization
FPA
L, nats
Surdden
add
mu1
add
LSTM-s2s att.
128
0.00
0.00
0.00
7.56
19.66
43.72
512
01.00
0.94
0.00
25.42
0.31
57.32
1024
0.25
0.75
30.29
5.26"
77.99
LSTM-s2s att.
0.00
0.00
15.32
17.37
45,09
0.00
30.26
1.40"
58.84
1024
0.00
1,00
0.00
51.70
3.09
86.82
CNN-s2s
128
0.00
0.00
1.00
281.34
301,75
0.02
512
0.00
0.00
1.00
318.12
346.19
0.00"
520.75
508.75
0.00
Transformer
35.21
46.07
2.94
512
0.00
0.00
1.00
38.77
50.64
3.50
1024
0.00
0.00
1.00
38.74
51.71
0.88
(b) Add-or Add-or-Multiply
nats linear
Syriddn
hiera
linear
hiezar
LSTM-s22s
0.00
0.00
30.40
79.00
512
0.05
0.00
31.04
61.84
1024
0.00
0.00
60.24
47.24
LSTM all.
128
0.00
0.00
32.72"
72.28
512
0.30
0.00
26.32
57.2
1024
0.05
0.00
65.80
73.80
CNN-s2s
128
0.00
0.95
178.88
0.00*
512
0.00
1.00
202.60
0.00*
225.36
0.12
Transformer
0.75
0.00
2,96'
36.92
0.69
0.00
4.84
35.04
1024
0.75
0.00
31.04
61.84
(c) Hierarchicallor-Lineees
FPA
L. pats
Syudton
comp
m:m
comp
L.TM-s22
128
0.00
0.00
37.67"
58.52
512
0.00
0.00
42.65
38.55
1024
0.00
0.00
37.66*
79.71
LSTM-s2s att.
128
0.00
0.00
66.65
63.84
512
0.00
0.00
62.34
70.92
1024
0.00
0.00
42.38'
73.28
CNN-s2s
128
0.85
0.00
0.99
$3.38
512
0.75
0.00
1.44
49.92
1024
0.80
0.00
1.24
$4.86
Transformer
128
0.00
0.66
155.50
6.23"
0.00
0.00
147.83
1024
0.00
0.60
151.70
6.02'
(d) Compesition-or-Memooiiiin Memorization
Table 4: Effect of the hidden size (SHidden): FPA measures the fraction of seeds that generalize according to particular rule. Description length L is averaged across examples and seeds. The lowest L are in bold and denotes stat. sig. difference in L < 10-2, paired t-test).
16

---

Published as a conference paper at ICLR 2021
FPA
I. nals
Sgmb
count
1.STMsx2s no att.
16
1.00
0.00
0.01
97.51
64
0.95
91.54
256
1.00
0.00
0.00
90.32
LSTM-s2s att.
16
0.99
0.00
7.84
121.88
64
1.00
117.39
256
1.00
0.00
9.79
127.31
CNN-22s
0.00
0.98
660.73
0.02
64
0.00
1.00
670.53
0.00
256
0.00
0.95
826.23
0.01
Transformer
0.00
116.34
11.10"
64
0.00
1.00
232.53
0.00
(a) Count-or-Memorizane Memorizatizn
FPA
L. nats
S Emdd
add
mul
mem
add
mul
mem
LSTM-s2s no att.
16
0.00
0.94
0.00
25.42
0,31
57.32
64
0.05
0.90
0.00
23.70
1.41"
51.33
256
0.05
0.84
20.42
1.33'
51.34
LSTM-s2s att.
16
0.00
0.98
0.00
30.26
1.40"
58.84
64
0.07
0.93
26.71
2.54
50.65
256
0.00
1.00
0.00
26.83
1,94
$1.20
CNN-s2s
16
0.00
0.00
1.00
318.12
346.19
0.00
64
0.00
0.00
1.00
293,86
294.50
0.00
256
0.00
0.00
1.00
486.81
447.20
0.00*
Transformer
16
0.00
0.00
1.00
38.77
50.64
3.50
64
0.00
0.00
1.00
87.11
142.83
0.00*
256
0.00
0.00
1.00
118.34
172.65
0.00
(b) Add Add-or-Multiphy
FPA
L, nats
8pmb
hierar
inear
bierar
1irear
LSTM-s2s no att.
16
0.05
0.00
31.04
61.84
64
0.00
34.44
72.2
256
0.00
0.00
36.32
76.56
LSTM-s2s
16
0.30
0.00
26.32
57.2
64
0.30
0.00
23.4
72.68
256
0.05
0.00
55,56
90.76
CNN-s2s
16
0.00
1.00
202.64
0.00
64
0.00
1.00
227.12
0,08
256
0.00
0.94
419.28
8.84
Transforner
16
0.69
0.00
4.84
35.014
64
0.00
0.00'
81.16
256
1.00
0.00
0.56
121.68
(c) Hierarchical-or--inn Linear
FPA
L, nats
Sgmn
camp
comp
LSTM-x2s att.
16
0.00
0.00
42.65
38.55
0.95
0.00
0.11
31.44
256
0.95
0.00
2.12'
23.63
LSTM-x2s att.
16
0.00
0.00
62.34
70.92
64
0.95
0.00
0.36
47.57
254
0.85
0.00
0.86'
49.39
CNN-22
16
0.75
0.00
1.44
49.92
64
1.00
0.00
0.01'
112.09
256
139.51
Transformer
16
0.00
147.83
6.36'
64
0.00
0.00
103.25
69.07*
256
70.76
107.54
(d) Composition-or cimiriooooo--Mmmmmiaaaaiiiio
Table 5: Effect of the embedding size (Smmb): FPA measures the fraction of seeds that generalize according to particular rule. Description length L is averaged across examples and seeds. The lowest are bold and denotes stat. sig, difference in (p < 10-2, paired t-lest).
17

---

Published as a conference paper at ICLR 2021
jump jimp around right tum twire
JUMP
RIURN JUMP RTURN JUMP KTURN JUMP ITTUNN TUMP ETTERN
jump opposite after walk aound
LTURN WALK
LTURN WALK LTURN LTURN JUMP
Figure Examples of SCAN trajectories and instructions, adopted from (Lkke & Baroni 2017
biased toward add whereas LSTM-s2s att, not show any significant bias with a slight preference for mem. In sum, the lower dropout is, the more likely learners will overfit the mem rule.
The same ohservation holds for the Composition-or" coririootoo--mmmiiztiiiin task in Table 6d That is, dropout has no impact on CNN-s2s and Transformer biases (the former show significant preference for comp while the latter prefers mem for any tested dropoul value), but does impact LSTM-based learners' biases, such dropoul 0.0 favors memorization.
Finally, we consider the Hierarchical-on or- -iriiic task (see Table 6c) We observe that, for any dropout value, CNN-s2s and Transformer inductive biases remain the same. Indeed, CNN-s2s learners show a consistent preference for linesr with FPA 0.75 while Transformers prefer hierar (note that this preference is not very large for dropoul 0.0 with FPA-hi cerrerrrraaa of 0.05, compared
FPA-hieras of 1.00 and 0.69 for dropout = 0.2 and 0.5 respectively). On the other hand, dropout has a larger impact on LSTM-based learners. When dropout- = 0.5, both LSTM-s2s prefer the hicrar hypothesis. However, for dropout = 0.0, both learners do not show any significant preference for any of the rules (with 0 FPA for both rules and close L values).
D SCAN
In the main text, we studied preferences of the learners towards compositional generalization in the Composition- or-Memorization task. Here, we resort larger-scale SCAN task (Lake & Haroni 2017). which can be thought of as a generalization of Composition-or" -oMooooria--oiiiioon with multiple modifiers. In SCAN, inputs are sequences that represent trajectories and outputs are step-wise instructions for following these instructions (see Figure 2 We experiment with the SCAN-jump split of the dataset, where the test set (7706 examples) is obtained by filtering all compositional uses of of the primitives, jump. The train set (14670 examples) contains all uses of all other primitives (including compositional), and lines where jump occurs isolation (only as a primitive). This makes the training data ambiguous only for the jump instruction. However, learners can transfer compositional rules across the primitives. We refer to this split simply as SCAN.
We consider two generalizations: (1) all input sequences with jump are mapped to single instruction JUMP as memorized from the training examples, or (2) underlying compositional grammar. We call them mem and comp, respectively. We used the original test as representative of comp candidate explanation and generated one for mem ourselves.
Recently, Hupkes et al. 2020) compared how different seq2seq learners perform on a context-free grammar, similar to SCAN, that requires compositional reasoning. Their work aims to study if and how learners pick up the underlying rules in the data. In contrast, our setup allows investigating learners' preferences. In particular, we apply description length metric to investigate learners' biases toward the mem and comp rules that explain the training examples in our setup.
Sequencce-t--sqquees learners We use the following models when experimenting with SCAN:
LSTM-s2s: We chose the architecture used in Lake & Baroni (2017) In particular, Encoder and Decoder
two hidden layers with 200 units and embedding of dimension 32. This architecture gets 0.98 test accuracy on the i.i.d. (simple) split of SCAN, averaged over seeds.
CNN-s2s: We use one of the successful CNN-s2s models of Dessi & Baroni (019)) that has 5 layers and embedding size of 128. We also vary kernel size in (3, 5, 8} as it was found to impact the performance on SCAN (Dessi& & Baroni 2019). These architectures reach test accuracy above 0.98 on the i.id. split of SCAN, averaged over 10 seeds.
Transformer: We believe our work is the first study of Transformer's performance on SCAN. For both Encoder and Decoder, we use 8 attention heads, 4 layers, embedding size of 64, FFN layer dimension of 256. This architecture gets 0.94 test accuracy on the i.i.d. split of SCCN, averaged over 10 seeds.
Training and evaluation We follow the same scenario described in main paper with two differences: (a) when calculating L. we use blocks of size 1024, (b) during training. we sample 10 batches with replacement, similar to Lake & Baroni 2017). We use batches of size 16 for LSTM-s2s & CNN-s2s, and 256 for Transformer. We repeat the training/evaluation of each learner 10 times, varying the random seed. Again, we use Adam optimizer and learning rate of 10-3. Due to the large number of test examples and low performance of the
18

---

Published as a conference paper at ICLR 2021
FPA
nats
dropont
count
ment
counL
mem
LSTM-52s
0.0
0.00
0.20
56.23
16.52
0.5
1,00
0.00
0.01
97.51
LSTM-s2s
0.32
0.2
0.95
0.05
33,66
87.36
0.0
0.99
0.00
7.84
CNN-2s
0.0
0.00
0.55
1034.67
0.43"
0.01
0.5
0.00
0.98
660.73
0.02'
Transformer
0.0
0.00
0.65
261 02
1,17
0.2
0.00
1.00
171.11
0.05'
0.5
0.00
0.97
116.34
11.10"
(a) Count-or- Memorization
FPA
L, nats
dropout
add
mul
add
mul
LSSM-s2s
0.0
0.25
0.00
0.00
5.00
35.55
19.07
0.2
0.30
0.45
0.00
12.07
11.04
37.39
0.00
0.94
0.00
25.42
0.31
57.32
LSTM-s2s att.
0.0
0.00
0.11
0.58
25.09
36.33
12.09
0.18
0.53
0.18
22.22
9,92"
34.48
0.5
0.00
0.98
0.00
30.26
1.40
58.84
CNN-s2s
0.0
0.00
0.00
0.65
236.41
247.52
0.42
0.2
0.00
0.00
1.00
438.06
464.26
0.00"
0.00
318.12
346.19
0.00
Transformer
0.00
0.00
0.65
84.58
130.88
0.96
0.2
0.00
0.00
1.00
65.62
99.05
0.02
0.5
0.00
0.00
1.00
38.77
50.64
3.50
(b) Aou-or-Multiply
FPA
dropout
hierar
linear
hierar
linear
LSTM-s2s no att.
0.0
0.00
0.00
16.38
17.72
0.00
0.00
11.09
19.87
0.5
0.05
0.00
7.76'
15.46
LSTM-s2s alt.
0.0
0.00
31.12
28.61
0.00
10.63
0.5
0.30
0.00
6,58
14,30
CNN-s2s
0.75
68.48
0.44"
1.00
99.42
1.16'
0.5
0.00
1.00
50.66
0.00*
Transformer
0.0
0.00
3,99
8.56
1.00
0.00
0.09'
13.09
0.5
0.69
0.00
1.21
8.76
Hlieaachhiallor-LLinear
FPA
L. nals
dropont
comp
mom
comp
mam
LSTM att.
0.2
0.30
0.00
6.42
11.01
42.65
LSTMsx2 all.
0.0
0.00
0.00
22.24
10.93*
0.2
0.45
0.00
4.38
35.14
0.5
0.00
0.00
62.34'
70.92
CNN-22
0.0
0.30
0.00
53.00
92.06
0.37'
0.5
0.75
0.00
1.44
49.92
Transforner
0.0
0.00
0.05
129.65
27.12"
0.2
0.00
0.00
102.90
27.56*
0.5
0.00
0.00
147.83
6.36'
(d) Composition-or- Momirioiioo--mmmmizaiaiiooooo
Table 6: Effect of dropout probability: FPA measures the fraction seeds that generalize according to particular rule. Description length L is averaged across examples and seeds. The lowest are in bold and denotes stat. sig. difference in L (p < 10-2, paired t-test).
19

---

Published as a conference paper at ICLR 2021
Accuracy
L, 1000 nats
comp
mem
comp
mem
LSTM-s2s no alt.
0.00
0.00
8.3"
45.0
LSTM-s2s att.
0.00
0.00
9.1"
34.8
CNN-s2s, kernel width 3
0.14
0.03
3.5
19.0
CNN-s2s, kernel width 5
0.26
0.02
2.0"
21.8
C2s- kernel width 8
0.36
0.01
1.8"
24.3
Transformer
0.00
0.00
7.3"
23.7
Table 7: SCAN:
Accuracy is averaged across seeds and examples. Description length L is averaged across examples and seeds. The lowest L are in bold and denotes stat. sig. difference in L. (p 10-3 paired t-test).
learners, all FPA scores would be cqual to zero. Hence, we follow Lake & Baroni (2017) and use per-sequence accuracy, i.c the ratio of the sequences with all output tokens predicted correctly.
We report our results in Table First, we see that CNN-s2s learners have a strong preference to comp, both in accuracy and description length. Furthermore, we observe that with an increase in the kernel size, deseription length of mem increases, while description length of comp decreases, indicating that the preference for comp over mem grows with the kernel width. We believe this echoes findings of (Dessi Baroni 2019).
Results
While accuracy well below 0.01 for other learner/ccaddidate combinations (and rounded to 0.00). according to the description length, Transformer and LSTM-based learners also have preference for comp over mem. This can be due to the transfer from compositional training examples, that can make comp explanation most "simple" given the dataset. Hence, the failure for systematic generalization in SCAN comes not from leamerr' preferences for mem. We believe this resonates with the initial qualitative analysis in (Lake Baroni 2017).
E STUDYING INDUCTIVE BIASES OF LM-LIKE ARCHITECTURES
While this paper we focused on studying inductive biases seq2seq architectures, we believe our entire framework can be used for investigating biases of architectures used in language modeling (LM). As language modeling emerges as general few-shot learning mechanism where textual prompting is used as an inter- face (Brown et al. 2020). it becomes important to understand how models generalize in such sctup. In this Section, we demonstrate the applicability of our tasks and measures to such learners and scenarios.
We propose to study inductive biases of LM-like architectures by using prompt-based versions our tasks akin to GPT2 (Radford el al.] 2019) & GPT3 (Brown et al. 2020). As an example, we can train an LM-like architecture to continue the sequence "<sos>aaa<sep=" as "ssos>aacsspp-bbbcco>>> and then prompt it with "<sos>aasa<sep>" and check the output: is it "<oos>anaccsp-bbbbee>ss> (mem) or ""xos>aatesee--bbbbooss (?ount)? Such formulation makes trivial adapt our tasks (Section 3) for the architectures used in language modeling. However, training differs from that a language model due the presence of the prompt.
In the following, we experiment with a variant of Transformer, namely joint source-targel sell-attention learner introduced by He et al. (2018). We use the code provided in (Fonollosa et al. 2019 The joint source source-target self-attention learner differs from the strueture of Transformer in three ways. First, instead of attending to the last layer of the encoder in standard Transformers, here, each layer the decoder attends the corresponding layer in the encoder. Second, the separate encoder-decoder attention module and decoder self-attention are merged into one attention, called mixed attention. The mixed attention is hence used to extract information both from the source sentence and the previous target tokens. Finally, joint source-target self-attention shares the parameters of attention and feed-for forward layer between the encoder and decoder. The of mixed attention and the sharing parameters make the encoder/decoder distinction artificial. Indeed, we can see this learner as a "decoder-only" LM-like architecture that is adapted to performing seq2seq tasks in a way we described in the previous paragraph.
We use the same hyper- prree-paraetters of Transformer presented in the main text. Preliminary experiments showed that joint source-target self-attention learners do not succeed the Hieearchiccll-rr-ineee Memorization lasks (i.e., less than 70% of the seeds learned the training set) when embedding size Semb 16 (the one used in the main experiments). We hence set Semb = 64. The remaining hyper-parameterss remain unchanged. Note that, according to Table 5 there is no switch of biases for any setting when considering
and Composition-or-
"oode link: bbttps://ee thub. com/ jarfo/joint However, unlike Fonollosa et al. 2019), we do not consider the locality constraint, and set the kernel 1ize__1 1 st parameter None. This makes the architecture equivalent the one introduced in (He et al. 2018
20

---

Published as a conference paper at ICLR 2021
Menirior-Meooiizaioonn
Add-or-Mullipiictiinn
Hicrarchical-or-Lineee
Compontiion-o--Meerrtaaai
count
mem
add
mu1
mem
hierar
lines=
ccmp
mem
0.00
1.00
1.00
0.00
0.10
0.00
L. nats
160.31
0.00
71,55
121.09
0
2.59
74.90
96.52
$7.94'
Table 8: Measuring biases of joint source-tareet self-attention learners: FPA measures the fraction of seeds that generalize according to particular rule. Description length I, is averaged across examples and seeds. The lowest I. are in bold and denotes stat. sig. difference in T. (p < 10-2, paired t-test).
Semb = 64 compared to Semb = 16. Specifically for Transformer, the only difference is that with Semb = 64, the latter shows even a stronger bias towards the hierarchical rule in the Hierarchical-or-Linee task.
Finally, similar Sections B and only consider the Count-or- or--eoorizzaiiinn task with ( 40, the Add-or- Multiply task with 20, Hierarchical-on -innearrr task with task with M = 36 and run 20 seeds per setting. Results are reported in Table 8
and the Composition-or -o-ioon--o---mmriaattiin
First, when considering the arithmetic tasks (i.c., Count-or-Memrriattn clear preference for the memorization rule (with a zero I. and 1.00 FPA) similar to Transformer behavior (see T) Second. joint pource-target self-attention learners are, similarly Transformers, biased towards the hierar rule with 100% of the seeds generalized perfectly to the hierar rule and a significantly lower Finally, for the Compositional-ov -orionnnnl-o---eeeee or-Memrrizaiinnn while only seed generalized perfectly to the comp rule, L shows a sigmificant preference for the mem rule, in agreement with Transformers' biases.
and Add-or--uuliplication) we note a
Overall, our experiments in this Section indicate that the inductive biases of the "decoder-only" Transformer learner very close that of a standard Transformer seq2seq learner. On higher level, we have demonstrated that our tasks and the description length measure can applied for studying inductive biases architectures that are often used in language modeling, indicating a high universality of the approach.
F EVOLUTION OF INDUCTIVE BIASES AT TRAINING
The description length measure that we defined by Eq. rrprresents both (a) how close the learner's 's generalization is to a fixed "explanation" of the data, and (b) if it can learn this explanation quickly, with few examples. In this Section, we aim peak into the latter process, by looking how quick learner can pick up candidate explanation. To do that, we use the Compositional ceci--rr-MMen -ol--oorimaatiion task with 1000 different primitives.
We organize the experiment in the following way. We start by training learners on the training data, composed of all primitives and M = 10 (i.c., 10 compositional examples). We measure description length of the comp candidate generalization on the remaining (1000-10 compositional examples). Next, train learners from scratch, but this time add 10 additional compositional example (M 10), and measure the description length on the remaining 1000-20 compositional examples. We repeat this procedure until all compositional examples are exhausted. As in this process the description length will decrease due to simply having less examples left for the evaluation, we normalize it by dividing by the number of remaining examples. We denote the result as L.
Since this is new task with larger number of primitives, the hyperparameters used in main experiments wouldn't 't allow the models reliably learn the training set. We hence use an embedding size of 64 for all architectures to have more robust convergence. For Transformer only, we also use larger architecture with layers. All the other hyperparameters remain unchanged compared to the experiments in the main text and we run 20 seeds for each setting. We report results in Figure
Figure shows that CNN-s2s are the fastest to prefer the compositional rule, starting already with low L. Indeed, when provided by only 20 compositional examples, is almost zero. On the other extreme, Transformers' behavior persists irrespective of M with large variance. While we note a decrease of L when increasing M, this tendency is very weak. Interestingly, while LSTM-s2s no att. learners start as less biased towards compositionality compared to Transformers, they display a lower L after seeing 120 compositional examples, and converge almost zero L with M > 600.
21

---

Published as a conference paper at ICLR 2021
3.0
2.0
LSTM
CNN--2s
0.5
Transformer
210
330
510
710
910
Number of compositional examples M
Figure 3: Per-example average description length (nats), across seeds for each learner function of number of compositional training examples. Shaded area represents 90% confidence interval.
22


Reviews:
[{"review_id": "w2mcM6WP5_l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to use description length (DL) to measure the seq2seq model's inductive bias. \n\nStrength: It is clearly shown that DL gives more smooth measurements than FPA. Also, the experiments are principled, and well designed. And the conclusion are interesting and clear. I do believe that this framework can be utilized by future work in this direction to get a better understanding of seq2seq models.\n\nFinally, the paper is well written, I am able to get the author's points.\n\nWeakness: My major concern is the scale or completeness of the experiments. The authors concentrate on a training set of a few samples, which is far away from the usual large-data setting for LMs. \n\nMoreover, the training data concerned only exhibit one or two kinds of bias, while in real data, there are usually various kinds of biases. I'm interested to see what the model will pick, when facing different kinds of structures in the data. For example, will the model favor easy rules than more complicated rules? \n\nIn addition to the different variants of seq2seq models, I'm also interested to see whether encoder-decoder model have different biases with pure LMs (only decoder, e.g., GPT-2).", "extra_scores": {}, "extra_text": {}}, {"review_id": "dsyIVtpaJlm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The Topic of this paper, to investigate the inductive biases of different neural network architectures, is super interesting. They do this by considering the extreme case of when there is only one training example and define a set of biases based on the type of solutions the model converge to when all of the options are equally optimal based on the given training example. \n\nThe authors claim that minimum description length is a sensitive measure of inductive biases. As far as In understood the idea is that a solution (rule) can be simple for one learner, while more complicated for the other one, based on their architectural differences. And by measuring the minimum description length of data generate by a certain rule under a specific learner, we can tell how simple the rule is for the learner.  I am a bit puzzled by the way this claim is phrased (I don't have anything against the main argument though). Isn't minimum description length itself an inductive bias that is probably applied on all these models? The way I see this is that a model that achieves lower description length is finding a simpler solution and the simplicity of the solution can be characterised by the different solution preferences that is defined in this paper (e.g. memoization vs counting). I would really appreciate if the authors provide a clear distinction between different sources of the inductive biases and their interactions in their experiments.\n\nA nice point about the paper is that in their experiments they evaluate the sensitivity of the preferences of different architectures to their hyper-parameters and it seems these preferences are consistent for most cases, but not surprisingly they observe that hyper-parameters like dropout rate can affects the generalisation behaviour of the models. That would be much nicer, if there were a bit more connections in the paper, about how these biases in the extreme case of only one training example cascade and affect the performance of the models when trained on larger scale datasets. For example, in a simple and still abstract case of gradually increasing the number of training examples, how the behaviour of these models change? Or in case of the point about few shot learning that is mentioned in the conclusion, I think it would be nice to have some experiments to show that the models will have these inductive biases and preferences even after being pre-trained on large datasets.  \n\nIn the end I agree with the authors that\n`Overall, our study shows that the inductive biases is more complicated than it seemed in these prior works and a more careful analysis is crucial.`\n\nAnd while I think quantifying inductive biases is a super interesting, challenging and important problem to address, this paper needs a bit more work to be more impactful in terms of either providing us with practical insights about the inductive biases of different architectures, or providing us with benchmark or tools to be able to fairly compare the inductive biases of different models, or just by more clearly identifying the main challenges for doing this in a beneficial way (e.g., because of the complicated interactions between different sources of inductive biases and how this all changes when the amount of data and the capacity of the models increase/decrease).  \n\n\n***\nPost Rebuttal: I like this paper and would vote for it to get accepted on the merits of: (1) Their finding about how MDL can be an indicator of inductive biases of the models; (2) introducing an experimental framework for studying inductive biases of the models. I'd also like to appreciate the authors' efforts to address reviewers concerns in the rebuttal. I agree with reviewer #5, that the paper can be better contextualised in the related research area but I think the paper is improved from this aspect a little bit during rebuttal and this is something that in general can potentially be fixed for the camera ready version. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BaEccdWKLSt", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a series of new datasets and task and investigates the inductive bias of seq2seq models. For each dataset, (at least) two hidden hypothesis could explain the data. The tasks investigated are count-vs-memorization, add-or-multiply, hierarchical-or-linear, composition-or-memorization. The datasets consists of one sample with varying length (amount of input/output pairs), which is denoted as description length. The models are evaluated on accuracy and a logloss. An LSTM, CNN, and Transformer are all trained on these datasets. Multiple seeds are used for significance testing. The results suggests that LSTM is better at counting when provided with a longer sequence, while the CNN and Transformer memorizes the data, but are better at handling hierarchical data. What this paper excels at is a thorough description of their experimental section and their approach to design datasets specifically for testing inductive bias, which I have not previously seen and must thus assume is a novel contribution.\n \nHowever, I lean to reject this paper for the following reasons\n- The paper tries to fit into the emerging field of formal language datasets for evaluating the capacity of deep learning methods. However, they do not build on any of the recent papers in the field. A new dataset, especially a synthetic one, should be well motivated by shortcomings of previous datasets and tasks in the field. I find the motivation and related works section lacking in that sense.\n- We already know that LSTMs can count https://arxiv.org/abs/1906.03648 and that transformer cannot https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00306\n- It is not clear to me why these results are important? Who will benefit from this analysis? Why are the current AnBnCn and DYCK languages that formal language people work with insufficient?\n- LSTMs do not have the capacity to perform multiplication. I don’t know why your results suggest otherwise. You would need to incorporate special units that can handle multiplications in the LSTM, such as https://arxiv.org/abs/2001.05016\n\nUpdate\n\nFirst I'd like to thank the authors for their detailed rebuttal. I have upgraded my recommendation from 3 to 4. As mentioned in my review I believe this approach is interesting. However, as pointed by reviewer2, the experimental section lacks completeness. I think this experimental section would be suitable for a workshop, but not a conference. I am excited to hear you are considering to use this method as an inspiration for real problems. I'd like to see the paper resubmitted when you have obtained such results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-P-qKUuyt2z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies inductive biases that are encoded in three main seq2seq architectures: LSTMs, CNNs, Transformers. Using one existing (fraction of perfect agreement) and one proposed metrics based on description length, and four dichotomy-like tasks, the authors show that, among other things, CNNs and Transformers are strongly biased to memorize training data, while LSTMs behave more nuancedly depending on input length and dropout. Unlike the first metric, the proposed metric takes values in a wider and more fine-grained range; the results are correlated for both of them. I also appreciate the attention to hyperparameter tuning and investigation of their effects in experiments. In general, the manuscript is well written and apart from a few minor questions can be accepted in its present form.\n\nQuestions:\n- SGD was found to often generalize better than its adaptive variants, like Adam (e.g. Wilson et al. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems. 2017), yet in your experiments there seems to be an opposite effect of changing the optimizer (Appendix C). Could you comment on why this is the case?\n- Regarding the tendency of large dropouts to hurt memorization: to what extend does this help the peer bias in a task? It seems that hindering memorization seem to cause a complementary increase in count or add-mul ability (Table 6). Is there a value for dropout (or a combination with other hyperparameters) when Transformers would start showing a counting bias?\n\nMinor:\n- please use alphabetic literature sorting\n\nUPDATE: I thank the authors for their detailed replies and running additional experiments. This resolves my questions and I'd keep my recommendation to accept the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a set of synthetic tasks to study and discover the inductive biases of seq2seq models.

Authors did a great job in convincing all the reviewers except R5 in their rebuttal. I do not find any serious concerns from R5's review. I personally think this is a very useful analysis paper.
Decision: Accept (Poster)