A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
Authors: Qing Li
Siyuan Huang
Yining Hong
Yixin Zhu
Ying Nian Wu
Song-Chun Zhu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Systematic Generalization
Concept Learning

Abstract: Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, HINT, to examine machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. Focusing on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t the three levels. Further, we design a few-shot learning split to determine whether or not models can rapidly learn new concepts and generalize them to more complex scenarios. To comprehend existing models' limitations, we undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results indicate that current models struggle to extrapolate to long-range syntactic dependency and semantics. Models exhibit a considerable gap toward human-level generalization when evaluated with new concepts in a few-shot setting. Moreover, we discover that it is infeasible to solve HINT by merely scaling up the dataset and the model size; this strategy contributes little to the extrapolation of syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting exhibits impressive results and significantly boosts the test accuracy. We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.%

Published as a conference paper at ICLR 2023
A MINIMALIST DATASET FOR SYSTEMATIC GENERAL- IZATION OF PERCEPTION, SYNTAX, AND SEMANTICS
Qing Li", Siyuan Huang', Yining Hong", Yixin Zhu", Ying Nian Wu², Song-Chun Zhu 1.2,3
1 National Key Laboratory of General Artificial Intelligence, BIGAI
"Cette for Vision, Cognition, Learning, and Autonomy (VCLA), UCLA
3 Instiiutee for Artificial Intelligence, Peking University
https: /ttiss -i- ustc. g git thub. 10/HINT
ABSTRACT
Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. In HINT, machines learning how concepts are perceived from raw signals such as images (i.e., percep- tion), how multiple concepts structurally combined form valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. Focusing on systematic generaliza- tion, we carefully design five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t. the three levels. Further, we design a few-shot learning split to determine whether or not models can rapidly learn new concepts and generalize them to more complex scenarios. To comprehend existing models' limitations, we undertake extensive experiments with various sequence- to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results indicate that current models struggle to extrap- olate to long-range syntactic dependency and semantics. Models exhibit a consid- erable gap toward human-level generalization when evaluated with new concepts in a few-shot setting. Moreover, we discover that it is infeasible to solve HINT by merely scaling up the dataset and the model size; this strategy contributes little to the extrapolation of syntax and semantics. Finally, in zero-shot GPT-3 experi- ments, the chain of thought prompting exhibits impressive results and significantly boosts the test accuracy. We believe the HINT dataset and the experimental find- ings are of great interest to the learning community on systematic generalization.
tasked with
INTRODUCTION
Humans possess versatile mechanism for learning concepts from data (Firestone & Scholl 2016). Suppose, for example, that we were tasked with deciphering ancient Egyptian signs based on the examples in Table I Given sufficient time, we may comprehend these signs by how recog- nize what each sign looks like at the perceptual level, by how to compose them into valid sequence- the syntactic level, and how to predict the results-at at the semantic level. Learning concepts heavily rely on these three-level interweaving meanings. Such observation is also con- sistent with the classic view of human cognition, which postulates at least three distinct levels of organizations in computation systems (Pylyshyn 1984).
Train
Test
Ldu 00 60
ou 18
f A $
b 100
Uere Tod 16
U Uu 12
Ji 41
=f-
A
mugn
8 2 (omeo > 26
Table 1: Can you decipher these ancient Egyptian signs from training examples and apply them to test cases? Interested readers can refer to the website, Egyptian] for more training and test samples with the ground-trune truth meaning for each sign. We strongly courage the readers to play this game prior to reviewing answers.
/tas:://liingg//// ustc. www.tggiitti......o 10/EINT/

---

Published as a conference paper at ICLR 2023
Another appealing characteristic of human concept learning is its systematic compositionality (Chomsky 1957; Montague 1970): the algebraic capacity to understand and construct an endless number of novel combinations from finite set of known components, ie.,, "infinite use of finite means" (Chomsky 1965.. As illustrated in Table 1] this form of compositionality is essential to the human ability to make strong generalizations from simple examples to complex ones.
Various benchmarks (Lake & Baroni 2018; Hupkes et al. 2020 Keysers et al. 2020) and methods
(Lake) 2019] Gordon al.) [2019] Csordás et al. (021) have been introduced by the emerging com- munity of learning models that capture human-like systematic compositionality. As it is difficult to collect real data with systematic compositionality. the majority of existing benchmarks are derived from artificial domains using synthetic data and tasks, covering only a subset of the concept learn- ing spectrum; see Table 2 for a detailed comparison. When evaluating systematic compositionalityy prior datasets frequently conflate syntax and semantics. For instance, the SCAN dataset (Lake & Ba- roni 2018) is a semantic parsing task from natural language commands to action sequences; when model fails longer command than the ones in the training set, the root cause could stem from misinterpreting the complex syntactic relations in a long input sequence (command) or its inability to generate long output sequence (actions) (e.g., as result of the EOS decision problem (New-) man et al., 2020). In addition, previous benchmarks frequently incorporated simple semantics (e.g., simple mapping repetition), resulting in undesired bias toward syntactic generalization.
To expand systematic compositionality to a full-spectrum systematic generalization w.r.t. percep- tion, syntax, and semanties, we draw inspiration from arithmetic and present new benchmark called HINT, Handwritten arithmetic with INTegers. The HINT task is intuitive: Machines accept as input images of handwritten expressions and predict the final results of expressions, restricted in the integers. Since there is no intermediary supervision, the three-level meanings are apparently inter- twined during learning, and models are expected to simultaneously acquire the three-level meanings to make correct predictions. To provide a comprehensive and rigorous test of how models generalize the learned concepts, we introduce a carefully structured evaluation scheme with five subsets, focus- ing on generalization patterns (i.e., interpolation and extrapolation) at various levels (i.e., perception, syntax, and semantics). In addition, we build a few-shot learning split to determine if models can rapidly learn new concepts from few examples and generalize them to more complicated scenar- ios. Being minimal yet comprehensive in terms of systematic generalization, HINT is fundamentally more difficult than earlier datasets because: (i) The images of actual handwriting with consid- erable visual variation; (ii) The syntactic relations between the tokens in the expressions are more complex with long-range dependency. (iii) The semanties of arithmetic concepts are more complex than the simple mappings in prior datasets.
To facilitate future research in this direction, conduct extensive experiments of various sequence- lo-sequence (seq2seg) models, including Recurrent Neural Networks (Hochreiter & Schmidhuberi
1997 Chung et al. 2014). Transformers (Vaswani et al., 2017), and GPT-3 (Brown al. 2020)
(with chain of thought prompting Wei et al. (2022)) ) Our experiments indicate that all models still struggle on HINT: even the state-of-the-ar model, Universal Transformer (Dehghani et al.) 2018)
with relative positional encoding (Shaw et al., 2018 Dai et al.) 2019). achieves just 54% accuracy on HINT, although it achieves virtually perfect accuracy on prior datasets such as SCAN (Csordás) let al., (02T). An in-depth analysis of the results on each test subset reveals that current models still struggle with extrapolation to long-range syntactic dependency and semantics. In the GPT-3 experiments, the chain of thought prompting significantly increases the zero-shot test accuracy from 8.6% to 27.6%. By examining the scaling trends of the test accuracy w.r.t. the size of the model and the dataset, find that it is impractical to solve HINT by simply scaling up the size of the dataset or the model, as is typically done in NLP tasks (Kaplan et al. 2020 Henighan et al. 2020): more data and parameters do not significantly improve the extrapolation over syntax and semanties. The few- shot learning experiments demonstrate that, despite the fact that the top pop-ffooominn models exhibit decent capabilities for learning new concepts, they are still far from the human-level generalization that only requires the learning examples of a new concept in a primitive form and readily generalizes to more complex compositions of the learned concept.
In short, we introduce the HINT dataset for investigating the systematic generalization across three levels percepperr,,,ion, syntax, and semantics. By benchmarking various seq2seq models on HINT, we uncover their primary weaknesses in systematic generalization. We hope the HINT dataset and our experimental findings will stimulate future developments of systematic generalization.
2

---

Published as a conference paper at ICLR 2023
Table Dataset categorization and comparison. SP: semantic parsing, IC: image classification, QA: question answering, i&t: image & text.
whether the task requires models learn percep- tionsyntav/semantics.. Generalization: the type of generalization required for test examples. *; the generated images in these datascts have little variance.
Dalasel
Domain
Task
Modality
Perception
Synlax
Semantics
Generalization
Size
SCAN
Lake Harom
2018
synihetic
SP
systematic
TODK
&SCA
(Ruus (Russ et ala
20211
synthetic
SP
18tt
systematic
300K
PCFG
Hiykes
synthetic
SP
systematic
100K
CFQ
Keysers ef al
A020
real
SP
text
systematic
239K
CURI
synihetic
IC
image
systematic
15K
COGS
Kim & Linzen 2010
real
SP
text
systematic
30K
Mathemancs
QA
systematic
2M
PGM
(Bateer air 2018
synthetic
IC
image
systematic
1.4M
CLOSURE
2019
synthetic
QA
i&t
systematic
7K
CLEVR
Jomnon Gf 2017
synthetic
QA
i&t
Lid
865K
HWF
image
Lid
MNIST-Add
MaMMBR cra 2018
real
IC
image
Lid
HINT (aurs)
QA
image
systematic
2
RELATED WORK
Benchmarks on Systematic Generalization Although several benchmarks
Lake & Baroni
2018 Hupkes et al.) 2020, Barrett et al., 2018, Zhang al. 2019 Teney et a 2020; Keysers et al. 2020 Bahdanau et al. 2019] Ruis et al.) 2020 Kim & Linzen 2020 Keysers et al. 2020) have advanced systematic generalization, the majority them are based on artificial domains with synthetic tasks, involve just one or two aspects of concept learning and often mixing the generaliza- tion over syntax and semanties. SCAN (Lake & Baroni 2018) tasked with translating a natural language command into a sequence of operations in simplified navigation domain using only syn- tax and semantics. CLEVR (Johnson et al. 2017) requires parsing questions (syntax) and grounding visual objects (perception), although objects themselves lack functional semanties. We refer readers to Table 2 for detailed comparisons of related datasets.
In contrast, the proposed HINT benchmark stems from the of arithmetic reasoning with real handwriting images (at the primitive level, rather than the expression level) and requires joint learn- ing of perception, syntax, and semanties. The precise definitions and boundaries of these meanings in HINT permit to build test splits to evaluate the specific generalizations. Notably, HINT possesses more complex semantics, which eliminates the undesirable bias towards syntactic generalization present in earlier datasets. The task of the HINT benchmark is inspired by the HWF dataset (Li al.) 2020) but requires full full- spectrum learning of perception, syntax, and semantics. By going be- yond an i.i.d train/test split in et al. (2020), HINT focuses on examining systematic generalization across many aspects of concepts.
Methods on Systematic Generalization To capture systematic generalization, new training regimes Lake 2019 Andreas 2020) Akyürek al. [020 | et al. (022) and model architec- tures Dessi & Baroni 2019 Russin al. [019 Csordás et al.) 2021 Gordon et al. 2019: Bergen ctt al. 2021) have been developed. Russin et al. (211)), for instance, expand a seq2seq model by segregating syntactic and semantic information. Csordás et al. (2021) investigate variety of Trans- former configurations to enhance its systematic compositionality, (nnrres (2020) and Akyürek et al.
(2000) investigate data enhancement for compositional generalization.
In particular, several neural-symbolic methods with domain-specific designs (Chen et al. 2020 Nye et al., 2020 Liu al. 2020) achieve near-perfect accuracy on prior systematic generalization datasets like SCAN (Lake & Baroni] 2018). However, these neural-symholic methods introduce certain non-trivial domain-specifie symbolic components, making it difficult transfer to other domains; their flexibility and transferability are unclear. In this paper, we benchmark on HINT with prevailing seq2seq frameworks, including RNNs, Transformers, and GPT-3, which require minimal domain-specific design and may be of broad interest to the learning community. We reserve for future research investigation of more sophisticated methods, such as data augmentation and neural-symbolic approaches.
3 THE HINT DATASET
In this section, we present the specifics the HINT benchmark, devised to evaluate models' capa- bility of learning generalizable concepts at three distinct levels: perception, syntax, and semanties.
3

---

Published as a conference paper at ICLR 2023
3.1 THE DEFINITIONS OF PERCEPTION, SYNTAX, AND SEMANTICS
We first define the perception, syntax, and semantics in the domain of HINT, as shown in Table 3 Perception refers to the mapping from image pixels into meaningful patterns, such as mapping an image of handwritten expression to symbolic sequence. Syntax refers to the mechanism of how the concepts in one sample are structurally organized e.g., parsing the symbolic sequence into a tree, and the syntax in Table 12 expressed by a phrase-structur grammar. Semantics refers to the functional meanings of these arithmetic concepts, e,g., what value '5' represents and what value '+' produces when given two arguments 1 and 1.
The definitions of perception, syntax, and semanties In syntax, number, opl, and op2 are the HINT grammar's 's pre-terminals in Table A2 semantics, and j are the operator's inputs. - is defined as max(0, i- j) to prevent negative results, and + is defined as ceil(i + j) to remove the decimal portions of the results.
Table 3:
main concepts
(b) new concepts in the few-shot learning split
concept
perception
syntax semanties
concept
perception
syntax
semantics
0..5..9
0.5.9
number parenthesis opl
0..5..9
x *X
number
11
none i+ max O, i 1 j) X j
y
y
number
12 max(i.j)
0
O
opl
opl
b
b
opl
min(i,
op2
op2
(i+
op2 ccil(i + j)
d o
op2
2i x xxx++(+0)
Notably, although these three levels have a clear boundary by their definitions, a model need not nec- essarily represent them by separate and individual modules. An end-to-end neural network trained on this domain, for instance, will likely contain neurons and parameters from all three layers. The notion of perception, syntax, and semantics simply requires the models to capture these meanings during evaluation, regardless of how the models finish the tasks, implicitly or explicitly.
Task The task of HINT is intuitive: predict the final results of handwritten arithmetic expressions in a weakly-supervised manner. That is, only the final results are given as supervision; all the sym- bolic expressions, parse trees, and intermediate values are latent. In such a setting, any model must simultaneously master perception, syntax, and semantics to solve this task successfully.
3.2
DATA GENERATION
Table 4: Examples from the training set and the test subsets of HINT.
Train
2X5+4 2
-4 (3-1xx(333-)-xxx00
(>-2421) 0
3x5--13--0-2 32
3x13333-2=-D--111
1:4 1 1x(8--6660 4-4+10
14/ 14/3-441 15
SS
[+3 3XCFX_)4()AA4H1)
66
6-10 (7-256)
0
LS
3X1 192 5703 51
135
XXC38/34 (4531196600X0000X660
438
Test
SL.
18
6
(7431/ (FS1)444-XK006660002444444444111
2
I.I.
(1021222444(11100330000:33312010 X1 (113((111111144111 D1t.H//000HDD////2000000000301.1
174
8 620
The data generation process consists of three steps. First, we extract handwritten images for each concept from CROHME (Mahdavi et al., 2019), including digits 0 through 9, operators +, -, x,, and parentheses (.) Second, we randomly sample prefix expressions and convert them to infix ex- pressions with necessary parentheses based the operator precedence; only single-digit numbers are permitted. The symbolic expressions are fed into a solver to calculate the final results. Third, we randomly sample handwritten images for symbols in expression and concatenate them struct the final handwritten expression. We only retain the handwritten expressions as input and the corresponding final results supervision; all intermediate results discarded.
Full-Spectrum Systematic Generalization To rigorously evaluate the systematic generalization of the learned concepts, we substitute the standard i.i.d. split with meticulously crafted evaluation
4

---

Published as a conference paper at ICLR 2023
scheme. We randomly divide all handwritten images into three splits: training (75%), validation (5%), and test (20%). First, we limit the maximum number of operators in the training set to 10 and the maximum intermediate values to 100:
Duain Turin
{( v) |x|| < 10, max(e) & 100),
where is the expression, [z]] its number of operators, y the final result, and all the intermediate values and the final results. To ensure diversity in the training set, we sample a maximum of 100,000 distinct expressions with the same number of operators. To prevent bias in the final results, we cap the percentage of a certain result at less than 5%. Next, we carefully curate the test set to evalu- ate different generalization capabilities (i.e., interpolation and extrapolation) meaning (i.e., perception, syntax, and semantics). Specifically, the test set comprises five subsets, formally defined as:
different levels of
Dust = 1o SS u LS u SL o LL, where generalization on perception only
(2)
C Duiing
SS = Timin' Dunin,
interpolation on both syntax and semantics
LS {(x, u) |a] 10, max(v) & 100),
extrapolation syntax and interpolation on semantics interpolation on syntax and extrapolation on semantics
SL {(x, v) |a] & 10, max(v)= > 100),
LL C ((2, |a|| 10, max(c) 1003. extrapolation both syntax and semantics
All subsets of the test set require generalization on perception since all images in the test set are unseen during training. For the test set, we sample no more than 1,000 unique expressions with the same number of operators, and the final results are also balanced. The maximum number operators is set up to 20, and the maximum intermediate value to 10,000. We also build a small validation set for hyperparameter tuning. See Table 4 for training and test examples and refer to Appendix A for further dataset statistics.
Few-shot Learning and Generalization To determine if models can rapidly learn new concepts, constructed few-shot learning split to learn six new concepts, shown in Table These six concepts have different meanings in terms of perception, syntax, and semantics: two new numbers (r *X and y y, representing 11 and 12, respectively), two operators of precedence (a O and b l. representing max and min), and two operators of precedence 2 (T. and d cf. representing arithmetic mean and harmonic mean). The train, validation, and test splits are constructed using the same strategy as in the full-spectrum generalization. Expressions are sampled to guarantee that the corresponding new concept appears at least once in the expression. This few-shot learning split is used to determine whether the models pre-trained on the training set can rapidly learn a new concept by fine-tuning on only handful of examples involving the new concept. In this context, "few-shot" implies that the examples used to acquire new concept are significantly fewer than those of the training set, but still exceed the number examples required by humans to learn a new concept.
DEEP
BASELINES
The task of HINT can be naturally formulated as input is a handwritten expression, segmented into a sequence of images by a sliding window, and the output is an integer, converted into sequence of digits. We benchmark deep seq2seq frameworks on HINT; see Figure for an illustration using detailed example.
(seq2seq) problem: The
4.1 IMAGE TOKENIZING AND EMBEDDING
Existing seq2seq frameworks typically accept a sequence of tokens as input. To tokenize a hand- written expression, its height is first resized to 32, and a 32-pixel sliding window is applied along the horizontal axis to render a sequence of images. Next, each image in the sequence is encoded by ResNet- 18 (He et al. 2016), sufficient to handle the visual variance in handwriting.
4.2 RECODER-DECODEE ARCHITECTURES
RNNs Recurrent neural networks (RNNs) have long been a dominant choice for sequence model- ing tasks. We test two popular RNNs in the literature: long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and gated recurrent units (GRU) (Chung et al. 2014). Each model is evalu- ated both with and without attention (Bahdanau et al. 2015).
5

---

Published as a conference paper at ICLR 2023
4
Encoder
Decoder
1
<SOS>
ResNet-18 18
<EOS>
SOSS>
1 0+5t
1
1
Figure I: The seq2seq framework applied an example HINT. ;SOS; start-of-sentence tokens. ;EOS;: end-of-sentence tokens. sliding window segments the handwritten expression into sequence of images, which are then separately encoded by ResNet-18 18. The expected output is a sequence of digits in reverse order.
Transformers Since its inception (Vaswani et al. 2017), Transformers have gradually supplanted recurrent or convolutional neural networks as the de facto choice for various sequence modeling tasks (Devlin et al. 2019; Radford al. 2019; Brown al. 2020). Nevertheless, prior work (De-
hghani et al. 2018] Hupkes et al. 2020 Kim & Linzen 2020) suggests that the vanilla Transformer fails substantially in many tasks requiring systematic generalization when the sequence lengths ceed those observed during training. Recently, several simple tricks have been proposed (Csordás) al., 2021) improve the generalization capability of Transformers;
of them work particularly
well: (i) using relative positional encoding (Shaw et al. 2018 Dai et al. 2019). and (ii) sharing weights across the blocks in the Transformer, a.k.a.., Universal Transformer (Dehghani al. 2018). Therefore, we benchmark Transformer variants: the vanilla Transformer, Transformer with relative positional encoding, and Universal Transformer with relative positional encoding.
Since the commencement of GPT-3 (Brown et al. 2020), there have been intense debates and different perspectives regarding the mathematical reasoning capacity of pre-trained large lan- guage models [ To systematically and comprehensively evaluate GPT-3's competence of arithmetic reasoning, we test it on the proposed HINT benchmark using symbolic expressions as input. Since all tokens of HINT are in the vocabulary of GPT-3, directly evaluate GPT-3 via shot prompting using the OpenAI API. We construct the prompt in the following form: "Q: What is Expression? A: The answer is". similar the practice in Brown et al. (2020). but with more complex expressions.
GPT-3
Recently, chain of thought (CoT) prompting Wei et al. 2022) has been extended to the zero-shot setting (Kojima al. 2022 by adding simple prompt. "Ler's think step by step," to facilitate step- by-step thinking prior to answering each question. Zero-shot CoT surpasses the standard zero-shot prompting by a significant margin in various reasoning tasks. Therefore, we also apply zero-shot CoT prompting to evaluate GPT-3 on HINT; we refer the readers to Appendix B.2 for the details of zero-shot COT.
4.3 TRAINING AND EVALUATION
Training All models are trained using the Adam optimizer (Kingma Bal 2014); the gradients ex- ceeding 5.0 are clipped. Dropout (Srivastava et al.) 2014) is applied to each recurrent layer of RNNs and each sub-layer of Transformers, including both the multi-head attention layers and the feedfor- ward layers. No training is required for zero-shot experiments on GPT-3; instead, 100 samples from each test subset are selected and fed to GPT-3 through zero-shot or zero-shot-CoT prompting.
Hyperparameter Tuning To produce reliable results, a thorough hyperparameter tuning is per- formed w.r.t. the number of layers in the encoder and the decoder, the dimension of the token em- bedding, the number of hidden units per layer, the number of attention heads in Transformers, the dropout ratio, and the learning rate. We refer the readers to Table A3 for further information.
Evaluation Metric We report the accuracy of the final results. A predicted result is considered correct only when exactly matches the ground-truth answer.
'Can GPT-3 do math? https://wwwyyuuuube..u http://wwwt.......////t///oot/e MMX.ANAVVV httpss:/opena: com/api/

---

Published as a conference paper at ICLR 2023
Table 5: The accuracy the test set using image inputs. All models jointly trained with randomly initialized ResNet-18. Reported accuracy (%) the median and standard deviation of 5 runs, "rel." denotes Transformer with relative positional encoding, and "uni." denotes Universal Transformer.
Model
Variant
88
1.
SL.
L.I.
Avg.
w/o att
61.3+1.4
53.3+1.7
30.5+ 1.2
9.2+0.2
11.9+0.
33.2+0.9
GRU
w/
66.7+-2.0
58.7+2.2
33.1+2.7
9,4+0.3
12.++100
35,9+1.6
att 80.0+:5.7
76.2+7.4
55.7+8.2
10.9+0.6
19.842.6
48.6++4.9
LSTM
w/ art 83.9+10.9
79.7+0.8
62.0+2.5
11.2
2..0
51.5+1.0
vanilla
20.9:10.4
9.340.2
5.740.3
1.540.3
2.9:0..5
8.345.3
Transformer
rel.
86.2.1..9
83.1113
60.112.3
10.9
19.400,5
51.7110
rel. uni. 8..4
86.011.
62.5
10.9
19.0
53.1
Table The accuracy on the test set using symbol inputs.
Model
Variant
1
SS
LS
SL
LL
Avg.
w/o att
1.69+
68.1
1.9
10.5+0.2
14.0+0.8
01.3++0.6
GRU
w/ att
76.2:10.6
69.5+006
4..8+1.5
10.5+0.2
15.1+1.2
42.5+0.7
wlo att 84.3+5.2
79.616.0
63.716.1
11.710.3
2..1
52.313.8
LSTM
w/ att 92.9+114
90.9+1.1
7..9+1.5
12.1+0.2
24.3++0.3
58.9+0.77
vanilla 93.9110.3
91.010.5
1.2
11.510.1
11.540.7
47.410.4
Transformer
rel. 96.6:1..3
95.110.4
1.5
11.8+0.2
22.340.6
59.4-10.5
rel. uni. 98.0++0.3
9.68+0.6
7..2+
11,7+0.3
22.4+1.1
61.5+0.9
0-shot 19.0
9.0
3.0
10.0
2.0
8.6
GPT-3
0-CoT 42.0
36.0
27.6
5 RESULTS
5.1
JOINT LEARNING OF PERCEPTION, SYNTAX, AND SEMANTICS
Tables and 6 summarize the results of all models on HINT using image inputs and symbol in- puts, respectively. Among all models, the Universal Transformer with relative positional encoding ("Transformer rel. uni.") has the highest average accuracy on the test set. Upon careful examination of the results, the following observations and insights can be made:
Models attains high accuracy on the subset Particularly, Transformer rel. uni using image inputs achieves an accuracy of 88.4%. The test subset shares the symbolic expressions with training and has different handwritten images for symbols. This indicates that Transformers and RNNs, jointly trained with 18, have strong generalization over perception. As depicted in Figure 2] the model forms meaningful clusters for each concept and captures syntactic roles to some extent without direct supervision on perception.
5
6
Figure 2: The t-SNE visualization of the embeddings (the outputs of ResNet-18) of handwritten images using the Transformer rel. univ. model. The image embeddings form clear clusters for cach concept based visual appearance. In addition, these clusters reflect the concepts' syntactic roles: The majority of digits towards the bottom, operators are around center, parentheses are near the top.

---

Published as a conference paper at ICLR 2023
Transformers achieve high accuracy the subset SS. The expressions in SS share the same length and value distribution as training. This result indicates that Transformers exhibit robust interpolation over syntax and semantics.
The accuracy of Transformer rel. uni. on LS is substantially lower than its accuracy on SS (see Table 6). Note that the identical model yields perfect accuracy on the length cutoff splits of SCAN (Csordás et al. 2021). This result, however rather unexpected, may be explained by the syntax difference between HINT and SCAN shown in Table A2 The expressions in HINT may have longer-range dependency and greater tree depth than the commands in SCAN. This obser- vation suggests that present Transformers, which have finite depth, are incapable of adequately capturing the syntax with long dependencies and large depth.
Transformer with relative positional encoding achieves similar performance on and SS as the vanilla Transformer with absolute positional encoding, yet relative positional encoding doubles the Transformer's accuracy on LS (see Table 6). This contradiction implies that relative positional encoding is essential for Transformer to generalize to long expressions. Sharing weights between the layers using the Universal Transformer can further enhance performance.
Models behave clumsily on the subsets SL and LL. The accuracy on SL and LL significantly lower than that on I and SS. All models exhibit near-zero accuracy samples whose answers are over 100 (the maximum final result the training set). This finding suggests that neither RNNs nor Transformers are able extrapolate to larger numbers beyond those in the training set.
While GPT-3 with zero-shot prompting performs poorly, chain of thought (CoT) prompting significantly improves the accuracy. Notably, GPT-3 with zero-shot CoT achieves accuracy of 49.0% on SL, which is superior to other fine-tuned models. We believe this is due to the fact that GPT-3 has been pre-trained on data with larger numbers, and CoT improves the reasoning process. Despite CoT prompting, GPT-3 performs poorly on long expressions in LS and LL.
Summary We observe a significant room for improvement on HINT. Even the best model, Univer- sal Transformer with relative positional encoding, only achieve accuracy of 54.3% on HINT, while the same model achieves virtually perfect accuracy on earlier datasets of systematic gener- alization, such SCAN. The challenges of HINT stem from the fact that it requires joint learning and generalization of perception, syntax, and semantics: The perception has a large variance real handwriting, the syntax supports long dependency between symbols, and the semantic complexity is well beyond the capability of the state-of-the-at models.
Scaling Laws Since HINT can generate an endless amount of data for training, one may wonder if merely increasing the dataset and the model size can solve the problem, akin to certain NLP tasks (aplaan al. 2020 Henighan et al. 2020). Empirically. Figure deppccs the test accuracy's ss scaling trend w.r.t. the model size and the number of training samples. By altering the hidden dimensions, the embedding dimension, and the number of attention heads, various-sized models are constructed. Similarly, various-sized training sets are generated by randomly sampling the original training set. Assuming a log-linear scaling trend, we need to train a model of 1033 parameters on 10.5 examples to attain 90% accuracy on the test subset LL, which is impractical. Hence, efficient architectures and training algorithms are still in need to improve extrapolation over syntax and semantics.
Model Size
OM
SM
10M
15M
20M
25M
30M
35M
50
Test Accuracy (Avg.)
abcd
Test Accuracy (LL)
20
45
15
40
35
30
20,
26.5)
5
Y 7.42 log 20.991
25
(2, 24.1)
log
0.0M
0.2M
0.4M
0.6M
0.8M
1.0M
2
10
Data Size
Operators
Figure 3: Scaling trends w.r.f. model size and dataset size when training Transformer rel. uni. the test subset LL with symbol inputs.
Figure 4: The few-shot learning performance when training Transformer rel. uni. with varied maxi- mum operators.
8

---

Published as a conference paper at ICLR 2023
5.2 FEW-SHOT LEARNING AND GENERALIZATION
In this section, we fine-tune the top two models on six new concepts: Table 7 summarizes the results. Transformer rel. uni. outperforms LSTM w/ attn across all concepts by significant margin, which is greater than six times their performance gap in Table 5 This discrepancy suggests that with limited data, Transformer is superior to LSTM learning new concepts.
Table 7: The fhw-shot learning performance of the top two models: LSTM w/ attn (left) and Transformer rel. uni (right). Reported results are the median runs. See [Table 3 for the meanings these concepts. *Please refer to Appendix Corr the details regarding the human study.
Concept
1
SS
LS
SL.
LL
Avg.
Human
87.8/992
47.3/80.2
42.8/58.6
10.8/12.2
16.4/19.3
42.8/52.8
95.0
y
64.5/83.8
39.1/74.8
38.5/54.0
11.6/13.8
18.9/22.4
35.4/50.7
100.0
0
71.8/84.4
44.2/72.0
29.7/48.9
7.9/8.4
11.1/12.3
33.8/46.4
97.5
b
73.4/77.1
29.9/59.1
27.4/39.4
7.4/16.8
12.7/17.1
31.1/42.6
77.5
c
61,5/59.2
19.6/34.0
15.2/24.4
4.5/6.1
6.5/9.4
21.9/27.3
90.0
d
59.2/62.8
22.7/39.0
20.2/27.0
7.2/9.2
8,9/10.7
24.7/30.4
60.0
Overall
69.7/76.1
33.8/59.9
29.0/42.0
8.2/11.1
12.4/15.2
31.6/41.7
86.7
Figure djeeiccss the test accuracy of Transformer rel. uni. while using varied maximum operators for training. general, the more data and longer expressions used for training, the higher the model's performance. One test case for learning new numbers ("ry") is (0, 26.5), where the model is only ex- posed to the primitive concept during training and expected generalize to complex compositions during testing. The classic thought experiments (Fodor 1975) indicate that this is straightforward for humans: If you grasp the meanings of "1," + 1," and "x;" you should also comprehend the meaning of "1 + x²". A similar test case for learning new operators ("abed"" is (2. 24.1) since ex- pressions comprising at least two operators are required to capture the syntax of a new operator.
Transformer performs poorly on both of these tasks, demonstrating that it is still far from human- level generalization.
6 DISCUSSIONS: CONCLUSIONS AND LIMITATIONS
In this paper, we took inspiration from arithmetic and introduced a new challenge for the learning community, Handwritten arithmetic with INTegers (HINT), which serves minimal yet com- prehensive benchmark for examining the full-spectrum systematic generalization of concept learn- ing w.r.t. perception, syntax, and semantics. HINT is intrinsically more challenging than previous datasets on systematic generalization due to its substantial perceptual diversity in real handwrit- ing, complex syntax, and sophisticated semanties. We benchmark HINT with the state-of-the-art seq2seq models, including RNNs, Transformers, and GPT-3; the results point out their inability to extrapolate over syntax and semantics. The scaling trends of test accuracy w.r.t. dataset size and model size indicate that it is impractical to solve HINT by only increasing the size of the dataset and model. We believe that the HINT dataset and our experimental findings will inspire new advances in systematic generalization, particularly extrapolation over syntax and semantics.
Limitations and Future Work Despite large visual variance, the handwritten expressions are rather basic in terms of spatial locations and visual complexity. It would be more intriguing if we could further increase the perceptual complexity w.r.t. spatial relations like natural images (Lin) ctt al. 2014). Although syntax and semantics in HINT are already more complex than those of prior datasets, they remain contexi-free. Extending our findings to context-dependent syntax and seman- ties would be of practical value given their prevalence in natural languages; e.g., word might have different syntactic roles or semantic meanings in different contexts.
Regarding model development HINT, our findings reveal that current seq2seq models, including Transformers, are unable to extract the systematic rules for both syntax and semantics from the training data. Improving the systematic generalization of Transformers, particularly extrapolation over semantics, is a crucial future direction. We also intend to investigate more advanced methods, such as meta-learning (Lake) 2019), data augmentation (Andreas) 2020 Akyürek et al.) 2020), Edge Transformer (Bergen et al.) 2021). and Seural-Symbolic Stack Machines (Chen et al. 2020). In addition, understanding the systematic generalization of large language models by evaluating them in few-shot or fine-tuning settings will be beneficial.

---

Published as a conference paper at ICLR 2023
Acknowledgements.
The authors would like thank four anonymous reviews for construc- tive feedback. This work is supported in part by the National Key R&D Program of China (2021ZD0150200) and the Beijing Nova Program.
REFERENCES
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. Learning to recombine and resample data for compo- sitional generalization. International Conference on Learning Representations (ICLR), 2020.
Jacob Andreas. Good-enough compositional data augmentation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In International Conference on Learning Representations (ICLR), 2015.
Dzmitry Bahdanau, Harm de Vries, Timothy J O' O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. Closure: Assessing systematic generalization of clevr models. In Visually Grounded Interaction and Language (ViGIL) Workshop NAACL, 2019.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference Machine Learning (ICML), 2018.
Leon Bergen, Timothy O' O'Donnell, and Dzmitry Bahdanau. Systematic generalization with edge transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. Compositional generalization via neural-symbolic stack machines. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Noam Chomsky, Syntactic structures. In Syntactic Structures. De Gruyter Mouton, 1957.
Noam Chomsky, Aspects of the Theory of Syntax. MIT press, 1965.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recur- rent neural networks on sequence modeling. In NIPS Workshop on Deep Learning, 2014.
Róbert Csordás, Kazuki Iric, and Juergen Schmidhuber. The devil is in the detail: Simple tricks improve systematic gencralization of transformers. Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer- xl: Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transform- ers. In International Conference on Learning Representations (ICLR), 2018.
Roberto Dessì and Marco Baroni. Cans found to jump around more skillfully than Compositional gen- eralization in seq2seq convolutional networks. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019.
Chaz Firestone and Brian Scholl. Cognition does not affect perception: Evaluating the evidence for "top- down" effects. Behavioral and Brain Sciences, 39, 2016.
Jerry A Fodor. The language of thought. Harvard university press, 1975.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivariant models for compositional generalization in language. In International Conference Learning Representations (ICLR), 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
10

---

Published as a conference paper at ICLR 2023
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom Brown, Prafulla Dhariwal, Scott Gray, et Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):173--1780)
1997.
Dicuwke Hupkes, Vema Dankers, Mathijs Mul, and Elia Bruni. Compositionality docomposed: how do neural networks generalise? Journal of Artificial Intelligence Rexearch (JAIR), 67:757-795, 2020.
Justin Johnson, Bharath Hariharan, Laurens van Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scolt Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2XX1::2001.0861,, 2020.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tihor Tihon, et al. Mcasuring compositional generaliza- tion: A comprehensive method on realistic data. In International Conference on Learning Representations (ICLR), 2020.
Najoung Kim and Tal Linzen. Cogs: compositional generalization challenge based semantic interpretation. In Annual Conference on Empirical Methods Natural Language Processing (EMNLP), 2020.
Diederik P Kingma and Jimmy Ba. Adam:
method for stochastic optimization.
arXiv preprint
arXiv: arXiv: 1412.6980, 2014.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Brenden M Lake. Compositional generalization through meta sequence-to--sqquence learning. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Brenden M. Lake and Marco Baroni. Generalization without systematicity; On the compositional skills of esquence-to-sequene recurrent networks. In International Conference on Machine Learning (ICML), 2018.
Qing Li, Siyuan Huang, Yining Hong. Yixin Chen, Ying Nian Wu, and Song-Chun Zhu. Closed loop neural- symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In Interna- tional Conference on Machine Learning (ICML), 2020.
Tsung-Y. Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dong- mei Zhang. Compositional generalization by learning analytical expressions. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2020.
Mahshad Mahdavi, Richard Zanibbi, Harold Mouchere, Christian Viard-Gaudin and Utpal Garain. Icdar 2019 crohme+ tfd: Competition on recognition of handwritten mathematical expressions and typeset formula de- tection. In International Conference on Document Analysis and Recognition (ICDAR). 2019.
Robin Manhaeve, Sebastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deep- problog: Neural prohabilistic logic programming. In Advances in Neural Information Processing Systems (NeurlPS), 2018.
Richard Montague. Universal grammar. Theoria, 36(3):773-38,, 1970.
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning. The eos decision and length extrapolation. In BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020.
Maxwell Nye, Armando Solar-Lezame Josh Tenenbaum, and Brenden M Lake. Learning compositional rules via neural program synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020.

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et Pytorch: An imperative style, heth-peffrmance deep learning library, In Advances in Neural Information Processing Systems (NewrIPS), 2019.
Zenon W Pylyshyn. Computation and cognition: Towards a foundation for cognitive science. Cambridge, Ma: MIT Press, 1984.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Jack W Rac, Scbastian Borgcaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 11446, 2021.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for sys- tematic generalization in grounded language understanding. Advances Neural Information Prucessing Systems (NewrlPS), 2020.
Jake Russin, Jason Jo, Randall C O' O'Reilyy, and Yoshua Bengio. Compositional generalization in a deep seq2seq model by separating syntax and semanties. arXiv preprint arXiv: 1904.09904 2019.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abili- ties of neural models. In International Conference on Learning Representations (ICLR), 2018.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In
North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15 (1):1929-1958, 2014.
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. V-prom: A benchmark for visual reasoning using visual progressive matrices. In AAAI Conference Artificial Intelligence (AAAI), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurlPS), 2017.
Ramakrishna Vedantam, Arthur Szlam, Maximillian Nickel, Ari Morcos, and Brenden M Lake. Curi: A bench- mark for productive concept learning under uncertainty. In International Conference on Machine Learning (ICML), 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoming in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, Song- Zhu. Raven: dataset for relational and analogical visual reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chun Zhu, and Yixin Zhu. Learning algebraic representation for systematic generalization in abstract reasoning. In European Conference on Computer Vision (ECCV), 2022.
A DATASET STATISTICS
The handwritten images for each arithmetic concept originate from the handwritten math symbols dataser" hosted on Kaggle under the "CCO: Public Domain" license, parsed and extracted from the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) (Mah-
davi et al. 2019 We further clean the dataset by removing duplicate images, resulting in statistics shown in Figure Al
We conduct detailed analysis of the collected data to demonstrate the validity of the HINT dataset as a benchmark for systematic generalization. Table AT shows the size of each split in HINT, and Table shows comparison between the grammars of HINT and SCAN.
https: http:///wwwwc.o cs. rit wwt....t....rrreee220111111
A1

---

Published as a conference paper at ICLR 2023
Table A1: Dataset size. The first is the main split of HINT, and the rest the few-shot learning split.
As advocated by [Csoráás (2021), the validation set also contains five generalization subsets model selection.
Test
Split
Train
Validation
Total
SS
LS
SL
LL
main
998000
4698
46620
9980
8000
10000
8640
10000
1100
491
4900
1100
900
1000
900
1000
100
493
4900
1100
900
1000
900
1000
1000
470
4700
1000
900
1000
800
1000
1000
470
4700
1000
900
1000
800
1000
C 1000
470
4700
1000
900
1000
800
1000
d
1000
470
4700
1000
900
1000
800
1000
Table A2: The phrase-structure grammars for HINT and SCAN. While the grammars of both HINT and SCAN can generate infinite examples. HINT produces examples with larger depth and longer dependency due the parentheses; the expression inside parentheses can be arbitranily long. Specifically, the maximum depth dependency range in SCAN are and 4, respectively; the maximum length generated by the non-terminal "S" the grammar of SCAN is 4.
HINT
SCAN
T= [Expression, Term, Factor, Number)
T= V.D.
Start symbol: Expression
Start symbol:
1, 9, (.)
(walk, look, run, jump, turn. left, right, around, opposite, and after, twice, thrice)
Expression - Term
==
Expression Expression Op1 Term
C --S S S and
after S
Opl ++
S S-V
twice V thrice
Term Factor
V + D[1] opposite D[2]
Term- - Term Op2 Factor
V + D[1] around D[2]
Op2 - x
V D U
Factor Number
D U left U right
Factor Expression )
turn left turn right
Number - 0] 0/.|2[3...19
U walk look run jump
For each split, we plot the frequency distributions of various aspects, including symbol, number of operators, expression length, tree depth, maximum dependency range, and result, as shown in Figure A2 The symbol distributions are similar across different splits, and the Kullback-Leibler vergence between train and test is low (0.0055). The digits and operators approximately equally distributed, except for the test-SL split. The test-SL split has a relatively higher portion of mul- tiplication (**') since generating large numbers generally requires more multiplication for short expressions.
The test set's result distributions differ from the train set. All results in the training set smaller than 100 as desired; about half are in [0, 10). In comparison, 29% of the results in the test set are larger than 100.
Several properties of an input expression, including length, number of operators, tree depth, and maximum dependency range, indicators of the difficulty of calculating the expression. We plot the frequency distributions w.r.t. these input properties in Figure A2 These distributions demonstrate significant differences between train and test.
A2

---

Published as a conference paper at ICLR 2023
6327
6210
6022
3978
3640
2724
2469
2403
1.95
1810
gpt
1649
1641
1194
1075
1027
648
B12
795
783
753
676
674
666
642
616
576
574
thetk
543
498
409
intty
alpha
369
355
348
321
1000
2000
3000
4000
5000
6000
Figure A1: The number of handwritten images for each symbol. There are 82 arithmetic symbols (the top 50 shown here) and 83,501 images in total. We the handwritten images for digits operators +, m, X, +. and parentheses (.) this work: others potential future use.
A3

---

Published as a conference paper at ICLR 2023
0.15
0.12
0.08
0.06
0.02
0.00
0.00
0.04
12
3.4
1.6
18
Number Operators
0.05
0.00
0.04
0.02
0.00
Expression Longth
0.30
0.25
0.29
0.35
0.19
0.05
0.00
0.14
0.32
0.08
0.81
0.67
12
15
16
17
20
23
22
2.5
25
Maximum Dapendency
0.3
0.2
0.1
0.0
1020
Figure A2: The frequency distributions w.r.. various aspects, including symbol, number of operators, pression length, depth, maximum dependency range, and result.
A4

---

Published as a conference paper at ICLR 2023
B
IMPLEMENTATION DETAILS
We benchmark deep sequence-to-sequeeee (seq2sea) frameworks on HINT, as illustrated by Fig-
ure
All models are implemented in PyTorch (Paszke et al. 2019).
B.1
IMAGE TOKENIZER AND EMBEDDING
To tokenize a handwritten expression, we first resize it by making its height 32 and apply a sliding window of size 32 along the horizontal axis to render a sequence of images. Next, each image in the sequence is encoded by the ResNet- 18 (He et al. 2016). We found in preliminary experiments that pre-training on the ImageNet does not help, likely due to the domain gap between ImageNet and HINT. Therefore,
a random initialization for ResNet-18 in our experiments.
B.2 ENCODER-DECODER ARCIITECTURES
We consider the following three choices for the encoder-decoder architecture in a seq2seq frame- work: Recurrent Neural Networks (RNNs), Transformers, and GPT-3.
RNNs
We test two popular RNNs: long short-term memory (LSTM) (Hochreiter & Schmidhuber
1997) and gated recurrent units (GRU) (Chung al. 2014). Both networks are evaluated with and without attention (Bahdanau et al, 2015 Our implementations of RNNs are adapted from a seq2seq tutorial
Transformers We benchmark three variants of Transformer: the vanilla Transformer, Transformer with relative positional encoding, and Universal Transformer with relative positional encoding. The implementations of these Transformers are adapted from Csordás et al. 22021 4
GPT-3 To test GPT-3's ability perform simple arithmetic operations without task-specific train- ing, Brown et al. 2020) developed a small battery of 10 tests that involve asking GPT-3 a simple arithmetic problem natural language; see Section 3.9.1 and Table 3.9 in Brown et al. (2020) for the results. In these tests, GPT-3 displays reasonable proficiency at simple arithmetic in the few- shot setting. However, they do not evaluate the multi-hop reasoning capability required by complex arithmetic expressions, which usually involve more operators and larger numbers.
To systematically and comprehensively evaluate GPT-3's capability arithmetic reasoning, we test GPT-3 on the proposed HINT benchmark using symbolic expressions as input. Since all tokens of HINT are in the vocabulary of GPT-3, we directly evaluate GPT-3 via zero-shot prompting using the OpenAI API) We construct the prompt in the following form: "Q: What is EExpression> A: The answer is," similar to the practice in Brown et al. 2020) but with more complex ex- pressions.
Via task-specific zero-shot or few-shot prompting, pre-trained large language models achieve ex- cellent performance in intuitive and single-step System / tasks Kahneman (201T). However, LLMs struggled on System 2 tasks that require slow thinking and multi-hop reasoning (Rae et al. 2021 at the scale of over 100B parameters like GPT-3. To address this shortcoming, chain of thought prompting (CoT) (Wei et al. (2022), which feeds LLMs with the intermediate step-by-step reasoning augment the final answer a few-shot setting, has been proposed to elicit the multi-hop reasoning in LLMs.
Very recently, chain of thought prompting has been extended to the zero-shot setting (Kojima al. 2022) by adding simple prompt, "Let' s think step by step", to facilitate step-by- by-pppp thinking before answering each question. Zero-shot CoT amazingly outperforms the standard zero- shot prompting by large margin in a variety of reasoning tasks. Therefore, we also apply zero-shot CoT prompting to evaluate GPT-3 on HINT. More concretely, follows a two-stage prompting strategy similar to Kojima et al. (2022):
1st prompt "Q: What is <Expression>? A: Let's s think step-by stypstep." prompt extracts the step-by-step reasoning process in the form of natural language from GPT-3,
" "This
https://qithin http:://gpenai.comm//
A5

---

Published as a conference paper at ICLR 2023
Table A3: Hyperparameter tuning. Our choices underlined.
Model
Variant
Encoder
Decoder
Embedding
Hidden
Heads
Dropout
Batch
Steps
Learning Rate
RNN
LSTM (+ all)
1,36,99
1,3,6.9
128, 256, 512. 128, 256, $12
0, 0.1, 0,5
128
100K
10 10 10
GRU
vanilla
Transformer
relative
1,36
1,366,9
256. 128. 256, 4,8,12
0.5
128
100K
10
100 10
relative universal
1.0
0.8
0.6
0.2
0.0
20
40
15
40
10
10a
length
depth
dippnndency
operators
result
Figure A3: Test accuracy (avg.) Transformer rel. uni. using symbol inputs as a function several prop- erties of samples: the expression's length, the depth the expression's 's parse tree, the expression 's maximum dependency range, the number of operators in the expression, the final result.
which is denoted by izz- 2st prompt "Q: What is <Expression>? A: Let's s think step-by-step. <2> Therefore, the answer (arabic numerals) is" In the second stage, the response iz, generated in the first step is appended to the initial prompt along with an answer trigger sentence. This second prompt is then fed into GPT-3 to predict the final answer.
In our experiments, we use the "text-davinci-00" engine in the OpenAI API, the most capable GPT- 3 model at the time of writing with approximately 175 billion parameter
B.3 TRAINING
Table A3 shows the tuned hyperparameters for the baselines. Our choices for each model are under- lined, and the performance is reported under these settings unless explicitly stated otherwise. When generating the output, we use greedy decoding in all models for simplicity.
For the few-shot learning experiments, models are first pre-trained on the main training set and then fine-tuned on the training set of each new concept individually. Models are fine-tuned for 1000 iter- ations using batch size of 128 with half examples from the main training set prevent forgetting. The learning rates are 10 and 10-3 for Transformers and RNNs, respectively.
All models reported in our paper can be trained on a single NVIDIA TITAN V GPU with 12G memory. It takes at most eight hours to train a model.
B.4 ADDITIONAL EXPERIMENTAL RESULTS
Figure A3 shows the test accuracy as a function of several sample properties. Figure A4 shows the importance of these properties.
C HUMAN STUDY FOR FEW-SHOT LEARNING AND GENERALIZATION
We conduct preliminary human study to evaluate human performance in the few-shot learning experiment. Specifically, we test ten human subjects on the six concepts that are unknown to subjects reduce the human prior as much possible. The human subjects asked determine each concept's 's meaning from 10 training examples and answer 4 test questions. We report the accuracy of test questions human performance.
'OpenAl API GPT-3 model sizes: httpsss//bboggggeeueee....d model si88
A6

---

Published as a conference paper at ICLR 2023
SS
LS
SL
LL
Importance
length
depth
dependency
operators
result
Figure A4: The importance sample properties w.rf. test accuracy Transformer rel. uni. using symbol inputs. Normalized permutation feature importance is reported here using a k-ncarest neighbors clas- sifier (k=3) to predict if the model can generate correct results.
A7


Reviews:
[{"review_id": "mAu9JsI4y4n", "reviewer": "Reviewer_LVqC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors created a large dataset that examine seq2seq models' ability to perform perception, syntax and semantics learning. The test sets are carefully constructed to study interpolation and extrapolation in various combinations. A large set of existing models are examined to check their performances.", "summary_of_the_review": "I like this paper and think this is above acceptance threshold. However, I'm waiting to see other reviewers's feedback (who probably are more familiar with the field than I am), before giving it a very strong endorsement.", "strength_and_weaknesses": "I want to preface this by saying that I'm not really familiar with this field.\n\nStrength:\n* The dataset is quite interesting to me. I think it can be a well-received dataset to test machine's ability to learn concepts.\n* The presentation is quite clear. The authors show several very informative tables (e.g. comparison with prior datasets, explanation of the various tasks). These tables are quite helpful for someone like me who's unfamiliar with the field.\n* The test set design is systematic and well designed.\n* A good group of existing models are examined. Informative results are provided. \n\nWeakness:\n* I don't know if it's standard practice, but as far as I can tell, the Egyptian characters are not part of the dataset? Not sure if that's an oversell of the paper. \n* the claim that \"Models show a significant gap toward human-level generalization\" is not experimentally evaluated. This sentence is briefly discussed in the caption of Figure 5, but i find the reasoning to be quite weak. The given dataset are based on simple arithmetic, so human are definitely having a very strong prior. By not evaluating the methods properly and isolate the effect of human prior, the claim seems unwarranted. For example, if everything is represented in Egyptian letter (perception level) and/or some new syntax / operations are involved, how can human extrapolate?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Zia2qAWvI8", "reviewer": "Reviewer_q55G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a visual reasoning task/benchmark that combines 3 levels: perception, syntax, and semantics. The reasoning aspect requires compositional generalization. Concretely, the task is to interpret handwritten arithmetic sequences, directly from images. Models are expected to learn all 3 levels end-to-end, without intermediate supervision. The evaluation is set up with different test set to measure performance of the various levels and with varying difficulty.", "summary_of_the_review": "I think this task/benchmark is at the right level of complexity for the current state of the art. I cannot see any major flaws in the paper and believe it will be of interest to the research community.", "strength_and_weaknesses": "### Strengths\n\n- Complementary to existing benchmarks(e.g. SCAN): realistic (non-synthetic) visual input, complex syntax and semantics.\n- Experiments with a broad range of models (classical seq2seq to modern transformers).\n- Various evaluation settings including a few-shot learning scenario.\n\n### Weaknesses / suggestions\n\n- W1: The existing datasets included in the review (Table 2) focus on those with training/test splits specifically designed to evaluate systematic/compositional generalization. It seems to me that other benchmarks for visual reasoning are also relevant to most other aspects of this work, such as the Raven-type matrices, for example [1,2,3]. [1,2] require perception and semantics (I believe), while [3] additionally requires perception of real visual inputs (entire photographs). I think this extension (perception requiring to parse entire photographs) is particularly relevant to this work (w.r.t. the first limitation - discussed by the authors - of the limited visual complexity of digits).\n\n- W2: The chosen task/domain has several limitations, but the authors already discuss these quite clearly at the very end of the paper.\n\n\n[1] [Measuring abstract reasoning in neural networks](https://arxiv.org/abs/1807.04225)\n\n[2] [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://arxiv.org/abs/1903.02741)\n\n[3] [V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices](https://arxiv.org/abs/1907.12271)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pbpz1EGR_g", "reviewer": "Reviewer_31D3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new language-like dataset, HINT, that requires a learning system to map handwritten arithmetic expressions to numbers. In contrast to prior datasets, HINT require learning systems to learn perception, syntax and semantics. Experiments reveal that current LSTM and Transformer based architectures are unable to generalize well on the problem, and scaling experiments indicate that an infeasible amount of data and parameters are needed to perform well on the task.", "summary_of_the_review": "The proposed dataset appears significantly more challenging than prior datasets, which is backed up by extensive experiments by the authors. The scaling experiments suggest that generalizing on this task will require new algorithmic or architectural innovations rather than merely larger scale. If this dataset encourages the development of new methods that can simultaneously learn perception, syntax and semantics, that could produce a very large impact in the field. Thus, I recommend acceptance.", "strength_and_weaknesses": "**Strengths**\nThe proposed dataset is simple to construct, yet very challenging which is a major plus point. As illustrated in Table 2, the dataset is categorically different from prior work since it tests systematic generalization over perception, syntax and semantics. The dataset has different levels of difficulty corresponding to different levels of generalization. Finally, a major strength of the dataset is the ability to test the acquisition of new concepts.\n\nThe empirical results are also well conducted. Strong baselines are tested on the task, including GPT-3, which provides confidence that the proposed benchmark is actually difficult to solve. The scaling experiments are also quite valuable since they demonstrate that the task can't be solved simply by using more data and computation. In my view, the few-shot learning results in section 4.2 are the most compelling experiments since they illustrate the difficulty of arguably the most challenging aspect of the task (namely, generalizing over semantics).\n\n\n**Weaknesses**\nOverall, the weaknesses of the paper are relatively minor. One concern is regarding the separation between syntax and semantics. In the proposed task, this separation is clear, but in other tasks it seems less clear. For instance, to what extent do solving tasks in CLEVR require understanding the semantics of objects in the scene? The authors appear to argue that semantics are not required, but justifying why this is the case would be helpful.\n\nAlso, the authors effectively test broad generalization over syntax and semantics. However, the way they test generalization over perception seems to be simply using different handwritten examples of the same symbols in the test set compared to the training set. Would it be possible to test a broader level of generalization over perception as well? For example, the authors could replace the symbols with the hieroglyphics from table 1 and test the learning system's ability to quickly learn and generalize to the new symbols.\n\nIt would also be helpful to understand how the models generalize separately on syntax and semantics. For instance, the authors may want to test generalization over syntax while keeping semantics fixed by testing on the same sequences as observed during testing but using different numbers (0, 1,... 9). This would be easier than the SS set proposed by the authors, but would still shed light on how the models operate.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "30jpGB0-_F", "reviewer": "Reviewer_52ig", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new dataset: HINT for studying machine reasoning across the joint domain of perception, syntax and semantics. HINT is a light-weighted dataset based on hand-written style digit arithmetic. Despite its simplicity, this paper introduces various generalization tasks over these three domains and demonstrates that state-of-the-art neural models perform poorly on these tasks. HINT sheds light on future studies of full-spectrum generalizability for machine reasoning.", "summary_of_the_review": "My current assessment to this paper is borderline accept, conditioned on the authors constructive response regarding related fields / baselines. I won't object if this paper is rejected because of weakness on these aspects.", "strength_and_weaknesses": "HINT is a well-designed dataset with simple data yet complex generation tasks. I like the minimalistic design to keep the generalization study clean across 3 different domains, this way we can diagnose a model in more depth. The paper includes evaluations of several state-of-the-art neural models, including variations of recurrent nets (LSTM, GRU), transformers, and GPT-3. Empirical study shows these neural models' capabilities of generalization across perception, syntactics and semantics, revealing a general lack of power on the latter two domains.This paper also provides insightful results on few-shot generalization of new arithmetic concepts.\n\nTo me the weakness of this paper is also obvious. The baselines are too simple and only concentrate on neural networks, while on the other hand semantic parsing has been long-studied by the NLP community ([1] as an example). Even within the domain of neural networks, there are models designed for generalization already been published for a long time (such as [2]). It is very unfortunate that these related works are missing from the discussion.\n\n[1] Guu et al. 2017 \"From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood\"\n[2] Andreas et al. 2017 \"Neural Module Networks\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%