From SCAN to Real Data: Systematic Generalization via Meaningful Learning
Authors: Ning Shi
Boxin Wang
Wei Wang
Xiangyu Liu
Rong Zhang
Hui Xue'
Xinbing Wang
Zhouhan Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: systematic generalization
meaningful learning
inductive learning
deductive learning
data augmentation

Abstract: Humans can systematically generalize to novel compositions of existing concepts. There have been extensive conjectures into the extent to which neural networks can do the same. Recent arguments supported by evidence on the SCAN dataset claim that neural networks are inherently ineffective in such cognitive capacity. In this paper, we revisit systematic generalization from the perspective of meaningful learning, an exceptional capability of humans to learn new concepts by connecting them with other previously known knowledge. We propose to augment a training dataset in either an inductive or deductive manner to build semantic links between new and old concepts. Our observations on SCAN suggest that, following the meaningful learning principle, modern sequence-to-sequence models, including RNNs, CNNs, and Transformers, can successfully generalize to compositions of new concepts. We further validate our findings on two real-world datasets on semantic parsing and consistent compositional generalization is also observed. Moreover, our experiments demonstrate that both prior knowledge and semantic linking play a key role to achieve systematic generalization. Meanwhile, inductive learning generally works better than deductive learning in our experiments. Finally, we provide an explanation for data augmentation techniques by concluding them into either inductive-based or deductive-based meaningful learning. We hope our findings will encourage excavating existing neural networks' potential in systematic generalization through more advanced learning schemes.

Under review as a conference paper at ICLR 2022
FROM SCAN TO REAL DATA: SYSTEMATIC GENER- ALIZATION VIA MEANINGFUL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Humans can systematically generalize to novel compositions of existing concepts. There have been extensive conjectures into the extent to which neural networks can do the same. Recent arguments supported by evidence on the SCAN dataset claim that neural networks are inherently ineffective in such cognitive capacity. In this paper, we revisit systematic generalization from the perspective of mean- ingful learning, an exceptional capability of humans to learn new concepts by connecting them with other previously known knowledge. We propose reassess models' compositional skills conditioned on the semantic connections between new and old concepts. In experiments, following the meaningful learning prin- ciple, we augment a training dataset in either inductive deductive manner to exposure such semantic links to models. Our observations on SCAN, as well two real-world datasets semantic parsing, suggest that modern sequence- to-sequence models, including RNNs, CNNs. and Transformers, can successfully one-shot generalize to novel concepts and compositions through semantic linking. We further demonstrate that both prior knowledge and semantic linking play a key role achieving systematic generalization and that inductive learning gener- ally works better than deductive learning. Lastly, we provide an explanation for data augmentation techniques by concluding them into either inductive- based or deductive-basse meaningful learning. We hope our findings will encourage ex- cavating existing neural networks' potential in systematic generalization through more advanced learning schemes.
INTRODUCTION
As crucial characteristic of human cognition. systematic generalization reflects people's ability to learn infinite combinations of finite concepts (Chomsky) 1957, Montague et al.) 1970p. However,
weak systematic compositionality has been considered as primary obstacle to the expression of language and thought connectionist networks for a long time (Fodor Pylyshyn| 1988] Hadley
1994 Marcus 1998 Fodor & Lepore, 2002 Frank et al.) 2009 Brakel & Frank 2009 Marcus
2018 Whether models can generalize systematically is still an appealing research topic until now.
Recent works state that modern neural networks have not mastered these language- based generaliza-
tion challenges in multiple explicitly proposed datasets (Lake & Baroni| [017]] Bastings et al.) 2018]
Keysers al. 2019: Hupkes et al. 2020 Kim & Linzen 2020). These studies conclude that models
lack such cognitive capacity, which calls for a more systematic study. Apart from the proposal of
benchmarks, existing research mainly focuses on novel architectural designs (Chen al. (0200) data
augmentation (Andreas) 2020 Akküüek et al. 2021) and meta-learning (Lake] 2019 Conklin et al.,
2021 to enable systematic generalization.
In this work, however, we question that whether neural networks are indeed deficient or just con- ventional learning protocols unable to exploit their full potential (Csordás al. 202I). Inspired by meaningful learning from the field of educational psychology (Mayer] (002).. we revisit systematic generalization to see whether neural networks still fail after semantic linking. To exposure semantic links between new concepts and existing ones, we augment prior knowledge through either inductive learning or deductive learning as what humans do in meaningful verbal learning (Ausubel) 1963). To be specific, inductive learning is bottom-up approach from the more specific to the mode gen- eral. By introducing new concepts sharing the same context with existing ones in specific samples, hope the model can capture the underlying semantic connections and thus generalize to novel compositions of new concepts. On the contrary, deductive learning is a top-down approach from the more general to the more specific. By involving rule-like concept dictionary without specific

---

Under review as a conference paper at ICLR 2022
Primitiive
Variants
Loxical Variant
Prompt
Incuctive Learning
Deductive Learning
Prior Knowledge
Semanttic Linking
New Compositons
jump right > TURN RIGHT JUMP:
jump > JUMP
Jump right - TURN RIGHT JUMP
jump ~ JUMP
jump-twice
JUMP JUMP
jump_ right TURN_RIGHT JUMP
jump n - JUMP
jump_0 - LURNLLEFT JUMP
jump_0
jump_0 0 twice JUMP JUMP
[jump] twice
....
jump_n left - TURN_LEFT JUMP
jump_n nn
jump n Niiee - JUMP JUMP:
Figure I: An illustration of the semantic linking injection pipeline in SCAN. The two middleboxes show the augmented dataset used for semantic linking through deductive learning (upper) and in- ductive learning (lower). In practice, the prior knowledge (left) and the augmented dataset (middle) are for training, and the new compositions of variants (right) are for testing. Models are expected to generalize to new compositions given prior knowledge and semantic linking.
context information, we hope the model can utilize the general cross-lingual supervised signals as anchor points so to launch the semantic linking. In experiments, inductive and deductive learning stand for training models on extra samples with or without context, respectively. as two standard data augmentation techniques (Arthur al., 2016 Wei & Zou| 2019 Nag et al. 2020). We mainly focus on three semantic relationships, namely, lexical variant, co-hyponym, and synonym. Starting from SCAN, our experiments confirm that, with semantic linking, even canonical neural networks can generalize systematically to new concepts and compositions. Moreover, this holds consistent across two more semantic parsing datasets. As an ablation study, we further examine such one-shot compositional generalization and find that both prior knowledge and semantic linking take essential parts. Lastly, we extend from toy sets to real data and explain how models' meaningful learning capability benefits them in solving real problems such machine translation and semantic parsing with the assistance of data augmentation techniques.
Overall, our contributions are follows: (1) We formally revisit systematic generalization by in- troducing semantic linking from a meaningful learning perspective. (2) We show how to conduct semantic linking by two common data augmentation approaches, either inductively
deductively. (3) We observe that modern neural networks can achieve systematic generalization with semantic linking, and both prior knowledge and semantic linking play key role, which is in line with mean- ingful learning theory. (4) We extend from SCAN to real data and demonstrate that many recent data augmentation techniques belong either inductive or deductive learning.
2 MEANINGFUL LEARNING
Learning new concepts by relating them to the existing ones is defined as a process of meaningful learning in educational psychology (Ausubel 1963] Mayer 2002). The utilization of meaningful learning can encourage learners to understand information continuously built on concepts the learn- ers already understand (Okebukola & Jegede 1988). Following the same idea, we intend to examine models' systematic compositionality by exploring semantic linking that establishes semantic rela- tions between primitives P (old concepts) and their variants V == {V, Vp E PP} (new concepts). To spoon-feed semantic knowledge to models for semantic linking, we augment the training data by either inductive learning or deductive learning (Hammerly) 1975 Shaffer 1989; Thornbury 1999) as humans learning vocabulary. In this section, we discuss the process of semantic linking and take "jump" from SCAN as an example primitive to illustrate the learning scheme in Figure]
2.1 SEMANTIC LINKS
We aim to revisit systematic generalization by exposing semantic links such as lexical variants, hyponyms, and synonyms. Lexical Variant refers to an alternative expression form for the same con- cept. Co-hyponym is a linguistic term to designate semantic relation between two group members belonging to the same broader class, where each member is hyponym and the class is a hypernym (Lyons & John 1995). Synonym stands for word, morpheme, or phrase that shares exactly or nearly the same semantics with another one. We provide an example and a detailed description in Appendix.
2

---

Under review as a conference paper at ICLR 2022
2.2 INDUCTIVE LEARNING
Inductive learning is bottom-up approach from the more specific to the more general. In grammar teaching, inductive learning is a rule-discovery approach that starts with the presentation of specific examples from which general rule can be inferred (Thorbbury 1999). Inspired by that, we propose to conduct semantic linking by introducing variant samples sharing the same context with their primitives during training. The assumption is that models can observe the interchange of primitives and their variants surrounded by the same context in the hope of coming up with a general hypothesis that there is semantic linking between primitives and their variants (Harris 1954). Formally, we describe inductive learning as follows. For a sequence-to-seque a source sequence I E X and its target sequence y E Y. We prepare prompts set Z := {z = fprompt (z) x E X) where fprompe() replaces the primitive in with slot mark [ Then, we generate X 1L := (x' = ffu((z,)) v) = E Z, E V by filling [zp] with variants in Vp. There is no change from the target side, so we get Y by copying y as y² for each x'L correspondingly. to operate semantic linking inductively. In practice, we first get prompt "[Zzump] twice" given primitive sample "jump twice". After that, we generate variant samples by replacing the slot mask "IZzumpl" with variants such as "jump_O _0". Finally, training models on generated variant samples like "jump_0 twice" combined with prior knowledge, we aim to establish the semantic relationships between primitives and their variants inductively. This process can be also treated as a kind of replacement augmentation (Wei & Zou. 2019).
task 7
X
+ Y. we have
Finally, we train models on ([ x
y ].
as shown in Figure [
2.3 DEDUCTIVE LEARNING
Deductive Learning, on the opposite of inductive learning, is a top-down approach from the more general to the more specific. As rule-driven approach, teaching in a deductive manner often begins with presenting& a general rule and is followed by specific examples in practice where the rule is ap- plied (Thombury 1999). To align with this definition, by combining a bilingual dictionary that maps primitives and their variants to the same in the target domain. This additional dictionary, hence, mixes the original training task with word translation Miikolov et al., 2013b). Without any specific context, we hope the model can utilize the general pross-lingual supervised signals as anchor points so as to launch the semantic linking. Formally, we describe deductive learning as follows. We first treat I' as the source dataset x directly and then prepare the corresponding target dataset YPPL by either decomposing samples from Y manually or feeding XPL to a trained external model. Similarly, we can consider V as another source dataset x.L and prepare its target dataset Y.P by copying the corresponding yp PL as y& DL for all x. as variants of each xPL After all, we get DLL as x_P and YDD as y,pL
intend to do semantic linking deductively
x.r.
(YPL The mapping from
X DL to Y DDL is a dictionary to translate primitives and their variants to the same targets without any specific context information. We name (x: y PT)) as a concept rule, (xp1. y PL) as a primi- live rule, and (n:24 y
variant rule since they are more rule-like without contexts. We train models on ([ _b [
Do 1) to operate semantic linking deductively, In practice, as presented in Figure D we directly make use of primitive "jump" and its variants such as "jump-0" as the source sequences with action "JUMP" as their same target sequences. By exposing both the primitive rule "jump" - "JUMP" and variants rules like "jump ("" "JUMP" during training, we intend to build the semantic connections between primitives and their variants deductively. This manner similar to the bilingual dictionary augmentation Arthhr al. 2016 Nag et al. 2020).
x
3 SYSTEMATIC GENERALIZATION
Although previous studies argue that neural networks fail to match humans in systematic gener- alization (Lake & Baroni 2017 Keysers et al., 2019), we revisit such algebraic compositionality conditioned on the semantic linking to see whether the conclusion will change. The following sec- tion moves on to specify the process and outcome of experiments. We first intend to make use of SCAN as the initial testbed to observe the presence of systematic generalization with the assistance of semantic relations. Then, we verify neural networks' potential to achieve the systematic general- ization activated by semantic linking on SCAN, as well as two real-world tasks of semantic parsing- Following ablation studies further examine models' compositional capability.
For each primitive, we pick only one prompt to fill in all its variants with Z specified for various datasets in Appendix.
3

---

Under review as a conference paper at ICLR 2022
SCAN
RNN
CNN
TFM
le5
1.00
Training Data Size
2.0
Test Seq. Acc.
1.5
0.99
1.0
0.98
0.5
0.97
100%
80%
60%
40%
20%
10%
10
8
4
1
Number Training Samples per Varlant
Figure 2: Experiments on SCAN with decreasing number of training samples per variant from the complete set (100%) to a single sample (1). The solid line represents the change of overall training size, and the dashed line stands for that of test sequence accuracy. There is hardly performance dip when training samples are deleted until only one remained.
3.1 DATASETS
There is evidence suggesting that SCAN may be far from enough to fully capture the kind of gen- eralization, where even simple model can behave as if it owns comparable skills (Bastings al.) 2018 Keysers et al., 2019). Thus, starting from SCAN, we introduce GEO and ADV generated re- spectively from real semantic parsing datasets: Geography and Advising Modification on datasets is specified in each experiment for the goal of examining machines' systematic generalization across various conditions.
SCAN is one of the benchmarks to investigate neural networks' compositional generalization (Lake Baroni] 2017). It includes 20910 10 pairs of commands English to their instructed action sequences "jump"" "look" "run", "walk") to be in line with previous works. We focus on lexical variants and create VSCAN by adding suffix that consists of an underline and a unique number for each primitive. We control |ySCAN| by setting the upper limit of this number. An example variant of "jump" is "jump_ 0" and both mean the same action "JUMP".
We define PSCAN
Geography is common semantic parsing dataset (Zelle & Mooney 1996 Srinivasan al. 2017) is also named as geo880 since it contains 880 examples of queries about US geography in natural language paired with corresponding query expressions. It later formatted to SQL. language with variables in the target sequences Frinegan-Doollas et al. 2018). GEO is the dataset generated based on Geography, where we regard 4 of 9 annotated variables as hypernyms and keep them as they are in SQL sequences. The other variables are restored by entities from the source sequence accord- ingly. As result, the overall data size 618 after processing and we can make use of the "is-a" hypernymy relations between entities and variables for semantic linking. To be specific, we define pCEO == ["new york city", "mississippi rivier", "dc", "dover") with yGEO consisting of entities as co-hyponyms sharing the same variable group with primitives An example variant of "new york city" "houston city" and both are the same variable group "CITY_NAME"
Advising, our second semantic parsing dataset, includes 4570 questions about course information in natural language paired with queries in SQL (Finegan- -Dolakk et al. 2018). Similar to GEO, ADV is generated on the basis of Advising with 4 of 26 variables hypernyms. Precisely, we define PÅDV = ={* {aa history of american film", ; "aaron magid", "aaptis", "100"} and VVDV as co- hyponyms of primitives sharing the same variables. For instance, "advanced at ai techniques" is co-hyponym of "a history of american film" sharing the same variable "TOPIC".
3.2 MODELS AND EXPERIMENTAL SETUP
What follows is account of network configurations and experimental settings. Without specific instruction, they are shared throughout experiments.
Models. After testing range of their adapted versions, employ three dominant model candidates with an encoder-decoder framework (Sutskever et al. 2014), that is, RNN, CNN, and TFM. In terms
https: : httpt/ttttbbbb com wccm/bbeenee wwwndennennnnnn www.ennnnlkkeeeeeemeeeece ake/ ake/SCAN SCAN
We select primitives for both GEO and ADV he align with SCAN.

---

Under review as a conference paper at ICLR 2022
of RNN, we reproduce bi-directional recurrent networks (Schuster & Paliwal 1997) with long short- term memory units (Hochreiter & Schmidhuber 1997) and an attention mechanism (Bahdanau et al. 2013). We follow the convolutional seq2seq architecture presented by Gehhing al. (2017) with regard to CNN and the attention-based structure proposed by Vaswani et al. (2017) ) in the case of TFM. More details are provided in Appendix.
Training. We apply the mini-batch strategy to sample 128 sequence pairs for each training step. We use Adam optimizer (Kingma & Ba 2015) with an & gradient clipping of 5.0 (Pascanu al.) 2013) and a learning rate of Ie to minimize a cross-entropy loss. We freeze the maximum training epoch at 320 for CNN and 640 for RNN and TFM. In contrast to early stopping (Prechelt 1998), we prefer a fixed training regime sufficient enough for models to fully converge in practice with a focus on the systematic generalization observation instead of superior structure exploration. To prevent uncontrolled interference, we train all models from scratch instead of fine-tuning (Devlin et al. 2019).
Evaluation. Token accuracy and sequence accuracy serve as two primary metrics in the following experiments. The former is soft metric that allows partial errors in a sequence, while the latter is tricky and strictly does not. The reported results, along with standard deviation, are the mean of five runs.
3.3 EXPERIMENT: MEANINGFUL LEARNING
This experiment probes the models' compositional generalization via meaningful learning in SCAN. We compare performances across various conditions starting from the conventional training pipeline as baseline. Usually, new concepts appear as vut-of-vocabulary (OOV). A typical training pipeline often involves replacement (Wei & Zou (019) to handle new concepts, especially for those sharing same or close semantic with existing concepts. Thanks to their incredible algebraic compositionality. humans effectively capture the underlying semantic connections between new concepts and old ones and generalize the prior knowledge to novel combinations by meaningful learning, given only a few demonstrations. To investigate the extent to which models can do the same, gradually reduce the number of training samples generated by replacement augmentation until there is only one for each variant. Although the final one-shot learning (Vinyals et al. 2016) is challenging, we hope to observe the presence of models' meaningful learning by measuring the performance loss due to a decreasing number of training samples per variant.
Experimental setup. Following replacement augmentation, we assign placeholders at the positions of primitives in source sequences and later put back their 10 variants but keep the identical target se- quences. Consequently, we have total of 329, 190 samples when |VSCAN| is 40 and randomly split them into training set (80%) and test set (20%). The training set is further processed to remove samples having multiple variants. Thus, we ensure that the number of variants' occurrences is while training at the one-shot condition. Eventually, the training dataset contains 235,002 samples. Models directly trained on this full dataset serve as baselines. Then, to format a gradual transition from baselines to mcaningful learning, we train the same models on various datasets conditioned on decreasing number of augmented samples for each variant until the one-shot learning setting. Besides, we use the variant rule "jump. 0" - "JUMP" as the only training sample for "jump O" as a case of our deductive learning and consider the rest our inductive learning.
Results. Surprisingly, as elaborated in Figure 2 RNN has no significant performance drop when the training size is reduced from 235,002 (100% to 16736 (1). It still achieves 99.92% test sequence accuracy when there is only one training sample for each variant. The same happens for CNN and TFM. Despite a slight fluctuation, they keep the results almost consistent regardless of whether the number of training variant samples is full or 1. The single sample works as whole augmented dataset and enables models generalize to novel compositions of learned variants. We want to un- derline that the sample can be either just variant rule or a variant sample derived from a primitive prompt. As we defined, developing semantic linking through the former is deductive learning, and that through the latter is inductive learning. By utilizing such semantic relations between primi- tives and their variants, models show they can perform one-shot generalization systematically via meaningful learning after semantic linking.
3.4 SEMANTIC LINKING INJECTION
Having observed the success after semantic linking, one question that needs to be asked, however, is how it works. Therefore, the following two experiments evaluate models' systematic generalization,
5

---

Under review as a conference paper at ICLR 2022
Table 1: Dataset statistics in Section Test size is often dozens of times the training size due to replacement augmentation. Additional details are offered in Appendix.
SCAN
GED
ADV
Expp
Exp.
Exp. IL
Exp.
Exp. IL
Exp.
Cha
St.
Ddd.
Cha.
St.
Sta.
Dif.
Cha.
Sta.
Size 209422
20928
20550
209466
720
711
728
734
6838
6084
3969
604D
Test Size 308240
308240
308240
305240
308240
21350
21350
21350
21350
21350
107614
107614
107614
101614
107614
particularly for prior knowledge and semantic linking. A sliding scale of difficulty is carefully designed by weakening these two factors according to the policy that the greater the difficulty, the more compositional skills are required. We further validate our findings on GEO and ADV. We use the same evaluation protocol across different datasets in this section.
Given dataset (X, Y) as prior knowledge regarding primitives, we generate the test set by replace- ment augmentation. Specifically, we replace the primitives in source sequences with their variants to generate novel compositions, So far, variants exist as OOV since they are not in the training set. Then, we incorporate additional either (X'1, Y or (X DT, Y the base training set (X, Y) so as to introduce variants in training and establish semantic linking inductively or deductively. As in the previous experiment, we ensure that each variant only has a single sample and appears only once. After training on ([ (1x [ vIL 1) ( xxx [ybz]) ]). models are evaluated on the same test set prepared before by replacement augmentation. For convenience, we keep the same settings as in the previous experiment, where |YSCAN| is 40 and 10 variants for each primitive. We use the full variants set for GEO, for example, 39 variants for "new york city". while we randomly sample 5 variants for each primitive in ADV so that we cover all the variants with an appropriate test size.
(XPL,
3.4.1 EXPERIMENT: SEMANTIC LINKING INJECTION VIA INDUCTIVE LEARNING
Experimental setup. We increase the difficulty of compositional learning by excluding primitive samples from the training set. We want to stress that, with higher level of difficulty, models have to generalize not only to new concepts but also to their new compositions.
Standard: Models are trained
([ ([*11 [T)x.) 1) without any adjustments.
Difficult: We remove primitive samples sharing the same context with their variant samples. For example, we remove "jump twice" due to "jump_0 twice", and thus models have generalize to "jump_0 twice" without seeing "jump twice".
We also exclude primitive training samples of the same length as their variant sam- ples. For instance, models have to reproduce the same generalization to "jump twice" without seeing primitive samples of length 2, including "jump twice", "jump right", "jump left" "jump thrice", and many others []
Challenging:
Results. In SCAN, what stands out in Table 2 is an excellent ohe-shot generalization for all three networks. The participation of (x y induces near-perfect generalization. Even the worst results obtained by TFM in Challenging are around 78% and 96.38% in terms of token accuracy and sequence accuracy separately. The outcomes confirm that networks can inductively learn the semantic relation from context after semantic linking. After that, models of different architectures can successfully achieve systematic generalization to novel compositions of variants during the test, What is noticeable is slight drop in both metrics the difficulty upgrades. The disappearance of the training samples in Difficult and Challenging settings can lead to a performance drop. This is well in line with the widely accepted belief in meaningful learning theory, well as our expectation, that prior knowledge is one of the keys related to humans' remarkable generalization. Therefore, conclude that both semantic linking and background knowledge exert powerful effects upon the potential of models to generalize systematically. The trends above on SCAN can also be found on GEO and ADV, while more apparent changes in metrics again verify our findings that prior knowledge is essential. Either excluding primitive samples containing the same context or those of the same sequence length as their variant samples can produce a steep fall in the generalization, which is not so sharp on SCAN. On GEO, CNN can lose an absolute sequence accuracy of almost 18. 18.26% from Standard to Difficult, and that for TFM drops 7.66%. This upholds our argument that generalization via meaningful learning is inseparable from sufficient prior knowledge. The overall
'We only remove samples that will not lead to unknown tokens.

---

Under review as a conference paper at ICLR 2022
Table 2: Evaluation results over RNN, CNN, and TFM on SCAN, GEO, and ADV in Section 3.4.1
conditioned on Standard, Difficult and Challenging settings.
Token Acc.%
Seq. %
Data
Model
Standard
Difficult
Challenging
Standard
Difficult
Challenging
RNN
99.99 + 0.08
99.89 + 0.19
99.96 0.02
99.95 0.08
99.85 + 0.08
99.80 + 0.31
SCAN
CNN
99.96 1 0.08
99.76 0.54
98.89 2.44 98.76
99.85 0.34
99.52 1: 1.07
97.57 5.24
TFM
98.91 + 0.78
98.90 = 1.10
0.85
97.35 1.62
96.86 + 2.64
96.38 2.81
RNN
75.71 + 8.42
75.69 + 6.12
73.46
3.05
44.95
14.69
43.27 = 13.47
36.77 + 5.60
GEO
CNN
87.99 1: 2.67
79.51 6.03
77.40 2.48 68.41
69 5.78
51:20 1: 8.64
48.58 3.40
TFM
75.37 1 7.84
75.11 1 4.88
4.76
45.93 12.42
44.59 1 9.76
36.93 1: 7.47
RNN
58.61 + 6.18
59.74 5.67
58.11
5.82
36.18 5.75
35.69 + 6.05
35,45 + 6.69
ADV
CNN
57.83 + 7.55
54.05 = 5.74
2.57 53.66
45.08 9.32
42.14 6.90
41.37 4.04
TFM
53.43 + 2.80
51.51 + 4.50
49.17 + 2.58
42.59 + 3.65
41.28 + 4.35
38.88 + 2.68
decline in performance can be attributed to the switch from toy sets actual datasets since both GEO and ADV own a much more complex encoding and decoding space than SCAN.
3.4.2 EXPERIMENT: SEMANTIC LINKING INJECTION VIA DEDUCTIVE LEARNING Experimental setup. We increase the difficulty of compositional learning by excluding primitive rules from the training set as follows:
Standard: Models are trained on (1x00]
y ( without any adjustments.
Difficult: We remove primitive rules from the training set, and train models on ( [ [x,p
Results. In SCAN, incorporating deductive semantic linking, all three networks are able to attain satisfying compositional generalization shown in Table 3 CNN achieves the highest 99.96% in Standard, while TFM takes the lowest 91.26% in Difficult with regard to sequence accuracy. How- ever, even the lowest one is impressive there is only during training. We can see a consistent decline in accuracy across three different models if we un- dermine the semantic linking by removing primitive rules in Difficult. The most significant sequence accuracy drop of 3.3% came from CNN when the difficulty upgrades. We further validate our find- ings on GEO and ADV, and find similar trend. There is persistent performance loss because of the absence of primitive rules from the training set across models. Concretely in GEO, the grade of CNN declines from 32.33% in Standard 23.58% in Difficult in terms of sequence accuracy. The causal role of semantic linking is also demonstrated by varying the difficulty. Overall, the joining of concept rules assists in developing semantic links between primitives and variants during train- ing, by which models can behave compositionally during the test. Moreover, the different outcomes between Standard and Difficult indicate that either concept rules or just variant rules can connect primitives with their variants semantically, though the former is better than the latter. Again, the overall performance fall is the result of the more complicated task. Another noteworthy finding is that neural networks can realize systematic generalization in either an inductive but perform better in the former setting. By comparing such preference in Table 2 and Table B we find that current black-box neural nets more capable of exploring rules and patterns from specific samples with context information rather than understanding knowledge from general concept rules in our experiments. This sheds light on why current machine learning is still highly data-driven and can hardly break through the bottleneck to realize advanced logic reasoning as human beings. How improve models' generalization in deductive learning an interesting future direction that we will work on.
variant rule to introduce each variant
deductive way
3.5
ABLATION STUDIES AND ANALYSIS
To explore other factors that may impact deductive learning, we conduct ablation studies with varying |N from {1 (1,2,344) and (YSCAN [from { (1,5.10,15,20) over RNN on SCAN. The ex- perimental setup is borrowed from Standard in Section 3.4.2
Impact of |PSCAN|. What attractive in Figure (a) is that when the number of primitives grows, the generalization performance improves simultaneously in terms of both accuracy boosting and variance reduction. It is counter-intuitive to see such improvement as we expect that primitive rules should work independently, and the number of primitives should not impact the systematic
7

---

Under review as a conference paper at ICLR 2022
Table 3: Evaluation results over RNN, CNN, and TFM on SCAN, GEO, and ADV in Section 3.4.2
conditioned on Standard and Difficult settings.
Token Acc.%
Seq. Acc. %
Data
Model
Standard
Difficult
Standard
Difficult
RNN
99.48 0.71
98.70 0.92
98.27 2.38
95.39 2.72
SCAN
CNN
99.99 + 0.01
98.59 + 3.10
99.96 + 0.03
96.66 7.27
TFM
96.90 + 1.78
96.68 + 2.21
91.91 + 4.04
91.26 + 5.80
RNN
54.44 + 7.15
39.71 + 18.38
13.61 + 7.08
7.76 + 5.34
GEO
CNN
41.86 + 3.38
41.07 + 7.48
4.85 + 4.66
4.04 + 2.18
TFM
67.02 + 6.91
65.97 5.17
36.38 10.08
31.57 + 7.42
RNN
36.50 + 7.66
36.42 + 7.39
12.84 + 4.31
12.66 + 5.19
ADV
CNN
43.51 11.31
35.34 14.68
32.33 12.93
23.58 16.04
TFM
56.82 + 3.79
53.33 + 3.85
47.43 3.71
43.24 5.14
(a)
1.0
Test Seq. Acc.
0.9
0.8
0.7
10 15
20
Number of Primitives
Number Variants per Primitive
Figure 3: Experiments over RNN on SCAN with varying |PSCAN| (a) and VSCAN (b).
generalization lot. potential reason is that semantic linking built by various "independent" primitive rules can profit each other to trigger a more robust and stable systematic generalization. For example, "jumpp" "JUMP" and "look" "LOOK" separate them from the samples with context information, such as "jump right" and "look right". Thus, we can regard "right" as compositional rules shared among primitive samples and finally encourage models to generalize effectively.
Impact of |VSCAN]. As presented in Figure 3 (b), RNN generalizes consistently well when the number variants goes up. Therefore, we report that the generalization among variants of the same primitive has certain degree of independence within reasonable range (e.g.. 20).
4 FROM SCAN TO REAL DATA
Thus far, have argued the feasibility of systematic generalization activated by semantic linking, as well as other factors such as prior knowledge. We move on now to discuss how such generaliza- tion already benefit the eventual performance of machines in solving real problems. Many recent papers propose to improve generalization on SCAN by data augmentation methods. Meta-learning is reported solve compositional problems by equipping models with memory loading (Lake) 2019). The success is reasonable due to augmented data for the application of meta-learning. By consider- ing concepts pointers in the memory, models are designed make connections between new and old concepts as semantic linking. Andreas (2020) suggests replacing fragments in real training sam- ples with others that sharing similar contexts, which can also be supported by our inductive learning. As in our findings, similar context information can help establish the semantic links between new concepts and old ones, thus enable models to generalize to novel combinations. Besides, we have proved how replacement augmentation (Wei & Zou 2019) works in Section 3.3 We would like to stress that the utility of similar unsupervised techniques (Xie et al., (2019) in both compositional generalization and real tasks can be attributed inductive learning as well since there need for systematic generalization in practice.
In addition inductive-basst based ones, notice many works incorporating bilingual dictionaries Arthhr et al. 2016, Nag et al. (2020), or called concept rules by us, in low-resource machine trans- lation can fall in the field of deductive-based methods. As proof-of-concept, reproduce the
8

---

Under review as a conference paper at ICLR 2022
Table 4: BLEU and SacreBLEU scores on IWSLT English-German (En-De) and German- English (De-En). IWSLT 15 English-Frenn French (En-Fr) and French-Englist (Fr-En) translation tasks. We mark the addition of concept rules as Vocabulary Augmentation
IWSLT
IWSLT
Model
Fa-De
De-Kn
Ea-Fr
Fr-En
SacreBI.EU
BLEL
SacreBI.EU
BL.U
SacreBL.RY
BLEU
SacreBI.EUI
Baselines
LSTM [Luong et
2015
24,98 27.39
24.88
30.18
32.62
38.06
42.93
37.34
39.36
Translomet
46,41
A0.45
42.61
Dynamic Cante Wirerai
27.28
33.33
35.54
40.41
45.32
39.61
41.42
+ Vocabulary Augmentation
LSTM (uong al 2015
25.351
25.387o.30
30.99%o.s
13.631..01
43.30f
57.77fm.41
39.83 39.837o.47
Transfomir Viswan
29.40/u.4s
29.297u.44
35.72"0.45
18.077
19
buzr
41,04fuum
43.157u.s4
Dynanic Cony.
Wu cral. 2019
27.60f as
27.507o
33.62 0.39
36.00 1o.00
40.87 16.47
45.95
39.951ma3
41.867o.4
Table 5: Token and sequence accuracy on Geography and Advising. We mark the addition of concept rules Entity Augmentation.
Geography
Advising
Model
Train
Test
Train
Test
Token Acc. %
Seq. Acc. %
Token Acc.
Seq. Acc.
Token Acc. %
Seq. Acc.
Token Acc. %
Sey. Acc.
Baselines
RNN
9.68
92.22
3.64
CNN
98.45
70.74 84.95
78.44 80.24
55,91
99.74
81.62
81.74 78.51
51.13
TFM
99.45
49.82
99.68
76.90
29.67
Enttity Angmentation
RNN
72.39f
15.05 15.051 a.
71.171
9a.95
CNN
97.54
76.03
80.32
60.93
99.65
87.01
84.50fa.75
56.027
81.09To.as
54.84fs.02
99.57
84.261
35.087p.41
word-to-word augmentation, or called deductive learning, by training models not only the base training set but also concept rules. Intuitively, we wonder to which extent deductive semantic linking can promote models' performance in common machine translation (IWSLT' 14 and IWSLT' 1 150) and semantic parsing (Geography and Advising). In machine translation, we construct bilingual dictio- maries by feeding vocabulary to the Google Translation API The word translation as concept rules if we consider synonyms of a primitive as their variants and such synonymous training supple-
be regarded
relationships as semantic links. Consequently, we get 144,874 word samples mentary for En-De and De-En, and 110,099 for En-Fr and Fr-En. In semantic parsing, we construct entity dictionaries by collecting entities (e.g.. "new york city"). They are translated to themselves since they do not change from the source natural language to the target SQL. The entity mapping can be treated as concept rules in the view of semantic linking. After that, we have a map of 103 entity translations for Geography and 1846 for Advising. A detailed experimental setup can be found in Appendix. We report the evaluation results in Table 4 and Table 5 where the same models with deductive semantic linking can consistently obtain performance gains.
5 CONCLUSION
Overall, we revisit systematic generalization from meaningful learning perspective and introduce semantic linking to exposure semantic relations between new and old concepts to models during training. To establish such semantic networks, we take advantage of two common data augmentation methods and name them as inductive and deductive learning to align with the meaningful learning theory. The observed one-shot generalization of modern machine learning methods can behave systematically after semantic linking. Extensive empirical results on SCAN, GEO, and ADV illustrate that prior knowledge and semantic linking two essential factors such generalization, in line with what humans do in meaningful learning- Given such findings, further group recent data augmentation methods in either the inductive- based or deductive-based category, followed by a proof-of-concce already benefits models in solving real tasks such machine translation and semantic parsing, We hope this paper can encourage future works to excavate neural networks' potential in systematic generalization through more advanced learning schemes.
SCAN supports that neural networks as class
to highlight how semantic linking
hhttps:/ccouu googg 1e www.co/t/aaaae

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
As far as we know, we do not see any potential concerns such as negative societal impacts from our work.
REPRODUCIBILITY STATEMENT
Code. processed datasets, and experiment log of this work can be found in the attached supplemen- tary materials and will be publicly available at GitHub. The source of raw datasets and external tools are presented in footnotes. Additional details regarding literature, model configurations, data statistics, and experimental setup are offered in Appendix reference in the main paper.
REFERENCES
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. Learning to recombine and resample data for compositionnlleeeenizaattooooo In International Conference on Learning Representations, 2021. UR//sttpss////peeerriiii http://wwww.eeetoooooaaaa/aaaaee IMPSc3kK
Jacob Andreas. Good-enough compositional data augmentation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7556-7566 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020 acl- aallmmain.676 URL https::::
//aclantho hoyeehhhoooy.....ooo //aallarcchooooyy.rr//2200...0.0 acl n. 676
Philip Arthur, Graham Neubig, and Satoshi Nakamura. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1557-1567, Austin, Texas. November 2016. Association for Computational Linguistics. doi: 10. 10.18///1//D16-11 1110066-116222 1162.
URL httts:://cclattho.l
brq/D16-1185
David P Ausubel. The psychology of meaningful verbal learning. 1963.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, 2015.
Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normal- ization and named entity recognition. In Proceedings of the Workshop on Noisy User- Eser-generated Text, pp. 126-135, Beijing, China, July 2015. Association for Computational Linguistics. doi: /0.18633/v/W11154330 URL htt.tttps:/// www.www.lww................ww... wwt//atto//aoooooooooooooooooo.t http://antolooo/ii..5333 4319
Fabian Barteld. Detecting spelling variants in non-standard texts. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Com- putational Linguistics, pp. 11-22, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https:::p///. www.wwlwwwl.............w.wwwwww wwt.:aaawwb..brr/cooooo.ooo////.
Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: SCAN both left and right. In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 47-55, Brussels, Belgium, November 2018. Association for Computational Linguistics. URL https: www. aclweb. oxc/antheloogy/llB--33
Philémon Brakel and Stefan Frank. Strong systematicity in sentence processing by simple recurrent networks. 3/th Annual Conference of the Cognitive Science Society (COGSC1-2009), pp. 1599-1604. Cognitive Science Society, 2009.
M. Cettolo, J. Nichues, S. Stüker, L. Bentivogli, R. Cattoni, and Marcello Federico. The iwslt 2015 evaluation campaign. In /WSLT7, 2015.
10

---

Under review as a conference paper at ICLR 2022
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam, volume 57, 2014.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. Com- positional generalization via neural-symbolic stack machines. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neu- ral Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, vir- Tual, 2020. URL
neur ips. hc//paper/222/hhasi (2blee22co1445222ff66622666 wtt.ee333611166077777033 Abstract.hir html
Noam Chomsky. Syntactic structures. 1957.
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. Meta-learning to compositionally gen- eralize. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3322 3335, Online, August 2021. Association for Computational Linguis- tics. doi: (0.18653/v1/2021 12213/11/222.....11o.2225533 lal.-lon....8..
URL
acl- ael-long.eo Long. 258
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. arXiv preprint arXiv:2108. 122844 2021.
Sarthak Dash, Md Faisal Mahbub Chowdhury, Alfio Gliozzo, Nandana Mihindukulasooriya, and Nicolas Rodolfo Fauceglia. Hypernym detection using strict partial order networks. Proceed- ings of the AAAI Conference on Artificial Intelligence, 34(05):7626-6633 7633, Apr. 2020. doi: 10. 1609/aaai. 334405.6263. URL hntpt:t::s pottps://o180 aaai org/ index.ndixxe php/ http:/AAA//artiiee view /2633 6263
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 20.18653/v1/N199 1423.
Catherine Finegan-Doll Dollak. Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasi- vam, Rui Zhang, and Dragomir Radev. Improving text-to-SQL evaluation methodology. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 351-360, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/P16 1033. URL /wwwwwww.......o/../o www.ee..gg/////boooo/o/ooooooo P18 -1033
Jerry A Fodor and Ernest Lepore. The compositionality papers. Oxford University Press, 2002.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analy- sis. Cognition, 28(1-2):3-7,, 1988.
Stefan I. Frank, Willem FG Haselager, and Iris van Rooij. Connectionist semantic systematicity. Cognition, 1103))358-379, 2009.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pp. 1243- 1252. PMLR, 2017.
Robert F Hadley. Systematicity in connectionist language learning. Mind & Language, 9(3)2447- 272, 1994.
Hector Hammerly. The deductionnindduction controveesy. The Modern Language Journal, 59(1/2): 15-18, 1975.
Zellig S Harris. Distributional structure. Word, 10(2-3)114-162,, 1954.

---

Under review as a conference paper at ICLR 2022
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip Yu. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-21, 2021. doi: 10.1 N..11099TNNSS.22211....
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compo- sitional generalization: A comprehensive method on realistic data. In International Conference Learning Representations, 2019.
Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based semantic interpretation. arXiv preprint arXiv:2010.05465, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL URL.http:
6980
William Labov. The social motivation of sound change. Word, 19(3):773-39, 1963.
Brenden M Lake. Compositional generalization through meta equence-to-ssquenee learn- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
Papera
Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compositional skills sequence-to-sequence
sequence recurrent networks. the 35th International Conference on Machine Learning (ICML 2018), 2017.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421. Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/1/DD5--1166 URL httphh::///// www.aclweb. lorg/ ogy pgy//11--166
John Lyons and Lyons John. Linguistic semantics: An introduction. Cambridge University Press, 1995.
Gary F Marcus. Rethinking eliminative connectionism. Cognitive psychology. 37(3):243-282, 1998.
Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2018.
Richard Mayer. Rote versus meaningful learning. Theory into practice, 41(4):226-22) 232, 2002.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scotisdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a. URL ARtpht//:/// wwwpiiiwwrggabbs.o.o 1301. 3781
Tomas Mikolov, Quoc Le, and Ilya Sutskever. Exploiting similarities among languages for ma- chine translation. arXiv preprint (rXiv:1309.44// 2013b.
Richard Montague et al. Universal grammar. 1974. pp. 222-46, 1970.
Sreyashi Nag, Mihir Kale, Varun Lakshminarasimhan, and Swapnil Singhavi. Incorporating bilin- gual dictionaries for low resource semi remi-supervised neural machine translation. arXiv preprint arXiv:2004 2020.
12

---

Under review as a conference paper at ICLR 2022
Dong Nguyen and Jack Grieve. Do word embeddings capture spelling variation? In Proceed- ings of the 28th International Conference on Computational Linguistics, pp. 870-881, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguisties. doi: 10.
18653/v1/2020.coling m0a.:oooiii---..55..... main.75. URL https: htmp:://www. ac www.we..... wegeb.org/anthholog///2222
Col man.ingmmin.5.
Peter Akinsola Okebukola and Olugbemiro J Jegede. Cognitive preference and learning mode as determinants of meaningful learning through concept mapping. Science Education, 72(4))489- 500, 1988.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-IILT 2019: Demonstrations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, ICML' pp. III-1310 11110--II---1 1318. JMLR.org, 2013.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pp. 186-191, Brussels. Belgium, Octo- ber 2018. Association for Computational Linguistics. doi: 10. 10.18633/1/W11--61199 URL
https: www ac
wwwp://wwwww.orgg...oo...o www.:wgw......................e. www..orggg....ogy.....ooo httplllly////111111111 6319
Lutz Prechelt. Early stopping- but when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 1998.
Herbert Rubenstein and John B. Goodenough. Contextual correlates of synonymy, Commun. ACM, 8(10):627-633 633, October 1965. ISSN 0001-0782, doi: 10.1145/365688.66666
hthps:::
orddi..org/0.. 1111/114536658 365657
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 ((11))2263-268) 1997.
Constance Shaffer. A comparison of inductive and deductive approaches to teaching foreign lan- guages. The Modern Language Journal, 73(4):395 1989.
Iyer Srinivasan, Konstas Ioannis, Cheung Alvin, Krishnamurthy Jayant, and Zettlemoyer Luke.
Learning neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 963-973, 2017.
URL http::/twwwtt aclweb. orgoo//////oooo wwtp///ttaoogg//eeeoooeoooeoeeee
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 2014.
Maja Stanojević et al. Cognitive synonymy: A general overview. FACTA UNIVERSITATIS- Linguistics and Literature, 7(2):193-200 2009.
Ilya Sutskever, Oriol Vinyals, and Quoe V Lee Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3104-3112 Curran Associates, Inc., 2014. URL httt:://pappp/:///e nips cc/paper/
Scott Thornuury. How to teach grammar, volume 3. Longman Harlow, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017.
13

---

Under review as a conference paper at ICLR 2022
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-366 3638, 2016.
Chengyu Wang and Xiaofeng He. Birre: learning bidircctional residual relation embeddings for supervised hypernymy detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3630-3640, 2020.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pp. 6382-6388. Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v/DD190 1/D11116110.. URL htttt:://accaaeetettttte
bxg/D19-1670
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. International Conference on Learning Representations, 2019.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 12848, 2019.
Jiale Yu, Yongliang Shen, Xinyin Ma, Chenghao Jia, Chen Chen, and Weiming Lu. Synet: Synonym expansion using transitivity. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: Findings, pp. 1961-1970, 2020a.
Yue Yu, Yinghao Li, Jiaming Shen, Hao Feng, Jimeng Sun, and Chao Zhang. Steam: Self-supervised taxonomy expansion with mini-paths. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1026-1035, 2020b.
John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intel- ligence Volume 2. pp. 1050-1055, 1996. URL haLhttp://// hwtp://llaacmm..coommmm wwt.org/ccaaai.in.ccoo
id=1 864516619... 9, 1864543
A APPENDIX
the following pages, we discuss our work in detail. Our code and data can found in the attached supplementary materials.
B SEMANTIC LINKS
Lexical Variant refers an alternative expression form for the same concept, where the various forms may derive from foreign languages, abbreviations, and even mistakes. A basic assumption is that all languages change over time due to non-lingui factors. Since the rise of sociolinguisties in the 1960s, studies linguistic variability, characteristic of language, are central to the language use and motivations for speakers to vary the pronunciation, word choice, or morphology of existing concepts (Labov) 1963). Taking "United States of America" as example, people have generally accepted the semantic connections among its lexical variants in history. including "America" and "United States", as well as the initialisms "U.S." and "U.S.A". Many efforts have been devoted on lexical variants representation (Nguyen & Grieve 2020), detection (Barteld 2017), normalization (Baldwin al. 2015) to keep machines up with the trend of the times.
Co-hyponym is linguistic term to designate a semantie relation between group members belonging to the same broader class, where each member is a hyponym, also called subtype or subordinate, and the class is hypernym (Lyons & John 1995). The "is-a" hypernymy relation
14

---

Under review as a conference paper at ICLR 2022
between generic hypernym and its specific hyponyms builds semantic connections among hyponyms. An example of such a hierarchical structure can be "Mississippi" and "Massachusetts" in the domain of "state". Specifically, "Mississippi" and "Massachusetts" are two hyponyms, and "state" is a hypernym. Thus, "Mississippi" and "Massachuseuts" are semantically connected to be co-hyponyms for each other. Harvesting hypernymy relations (Wang & He 2020) plays an essential role for downstream knowledge graph construction (Ji et al. 2021 out-vocabulary generalization (Dash et 2020), taxonomy expansion Yu et al., 2020b), etc.
Synonym stands for word, morpheme, or phrase that shares exactly or nearly the same semantics with another one. Many tend assume synonyms are utterances that occur in most contexts in com- mon, so they are semantically closely related enough to be synonyms for each other (Rubenstein & Goodenough] 1965, Harriss 1954). The existence of the association to contexts a basic assumption supporting the advance of recent masked language modeling (Devlin et al. 2019). Given that, one of the definitions of a synonymous relation semantic link between two expressions substitution of one for the other never hurts the true value of the context (Stanojević et al. 2009). For instance, the substitution of "heavily populated" for "populous" will seldom alter the truth of the sentence in Figure 4 Such semantic similarity representation as well (Mikolov et al.] (2033)). Synonym discovery (Yu et al.) (2020a) has been a fundamental job construct knowledge base and thus benefits substantial researches.
be observed in continuous vector space from trained
Lexical Variant Synonym Antonym Co-hyponym Hypernym Hyponym
state
uninhabited
area
U.S.
heavily populated
congested
Mississippi
Massachusetts most populous state the New England region of United States
Figure 4: A concrete example of semantic linking. The bidirectional arrows denote symmetric relations. Mississippi and Massachusetts are two specific states, thus both hyponyms of state. In turn, state is a hypernym of them. Due to a common hypernym, Mississippi and Massachuseits become co-hyponym for each other. (heavily populated, congested, populus), {area, region). two groups of synonyms for sharing same similar semantics. Finally, U.S., kind of abbreviation, is a lexical variant of United States.
C DATA
IWSLT involves IWSLT'14 14 (Cettolo et al.) 2014) English-German (En-De) and German-Englist (De-En), IWSLT' (Cettolo al. 2015)) English-French (En-Fr) and French-English (Fr-En) trans- lation tasks. The goal is to translate sentence from one language to the other. The IWSLT' 14 En-De and De-EN have 160,239 sequence pairs for training and 7,283 for validation. We make use of IWSLT14.TED TED. (2010) 2012] and IWSLT14.TED tsttttt20000,,, 2011, 2012] to measure trans- lation performance. resulting in a total of 6,750 test samples. In terms of IWSLT' 15 En-Fr and Fr-En, there are 205,572 sequence pairs for training. We employ IWSLT15. SEE.15.TDD.CCM THD.dev201 and IWSLT15.TED.ss 22010000, 2011, 2012, 2013) the validation set and (WSL115.ss(200 as the test set. As consequence, there are 5,519 samples for validation and 2,385 for evaluation. For all four translation tasks, we apply BPE with 10K tokens to share.
2015)
D MODELS
All models are developed with the encoder-decoder framework (Sutskever et al., 2014). We repro- duce RNN, CNN, and TFM by ourselves to have fewer parameters than original versions for the convenience of verifying systematic generalization. The dropout rate is 0.5 for RNN, CNN, and TFM (Srivastava al. 2014). We implement LSTM, Transformer, and Dynamic Conv. under the
15

---

Under review as a conference paper at ICLR 2022
Table 6: Example source and target sequences from SCAN, GEO, ADV, Geography, and Advising.
Data
Sequense
Source
Jamp noice
Target
TUMP TUMP
how mun people new
Target
SELECT
POPULATION FROM CTTY
CITY NAME CITY NAME:
ADV
Souce
Which depurdment
DISTINCT COURSE alus0. DEPARTMENT FROM COURSE AS COURSE alsod WHERE COURSE
fim
LIKE TOPIC;
Geography
Source
live york SILECT STATE aloso, POPULATION FROM STATE AS STATE aliaso WHERE STATE aliso SAATE nnw
how
Soure
I would liv
Hosorry Amerioon Film cosmmes crmilss
SBI EELECT DISTINCT COURSE aliasl DEPARTMENT COURSE alsxd NAME, COURSE alizoll. NUMBER FROM
Advicing
Taget
COURSE AS COURSE dlall WHERE COURSE aliao. DESCRIPTION "IKE History of American Film 8"
OR COURSE aliao. NAME LIKE 1% History of American Film 8") AND COURSE alimd) CREDITS 22;
framework fairseq Out et al. 2019) and inherit its default model structures Without notes in tasks, hyperparameters are shared throughout the work. We train all of our models on a single Nvidia Tesla V100.
RNN denotes bi-directional recurrent network (Schuster & Paliwal 1997 Hochreiter & Schmidhu-
ber 1997) with long short-term memory units and attention mechanism (Bahdanau al. 2015)) Its encoder consists of two layers with hidden size of 256 in each direction, and its decoder has one layer with a hidden size of 512. The embedding size 512 for both encoder and decoder. There are a total of 5. M trainable parameters. Teacher forcing with rate of 0.5 serves to spur up the training process [Williams Zipser 1989).
CNN denotes the fully convolutional seq2seq network (Gehring et al. 2017). The size of the position embedding layer is 128 for encoding and 256 for decoding, while that of the token embedding layer is 512 for both encoding and decoding. There are 10 convolutional layers with 512 the hidden size and 3 as the kernel size in both encoder and decoder to generate a total of 33.55M trainable parameters.
TFM denotes transformers, an attention based network (Vaswani et al. 2017). As a tiny version, TFM has 2 layers for each encoder and decoder with 8 attention heads and dimension of 512. The size of the feedforward layer is 2048. We utilize the cyclic nature of sin and cos functions to represent token positions. There total of 15.02M trainable parameters.
LSTM is adapted from the recurrent network used by Luong et al. (2015) for statistical machine translation. The size of the embedding layer 1000. There are 4 layers both encoder and decoder with a hidden size of 512 and a dropout rate of 0.2.
Transformer, the same as TFM, is adapted from the base version of transformers in the work of Vaswani et al.] (2077), while TFM is tiny version to test systematic generalization. The dimension 512 for the embedding layer, 1024 for the feedforward layer, and 512 for the attention layer. There are 6 attention blocks in both encoder and decoder with 4 attention heads and 0.3 dropout probability.
Dynamic Conv. is adapted from the seq2seq convolutional network proposed by Wu et al. (2019), where the hidden size of the embedding layer, encoder layer, and decoder layer is 512. The number of attention heads is 4, and the dimension of the feedforward layer is 1024 for both encoder and encoder. There are 6 layers the encoder and 7 layers in the decoder. The dropout rate 0.1 for both attention and weight units.
E EXPERIMENTS
SEMANTIC LINKING INJECTION VIA INDUCTIVE LEARNING
Semantic linking can be operated via inductive learning, where replace the concept in the prompt with primitives and their variants. The learning rate to train CNN in GEO is changed to Se
https: thub. com www.//yyoocc/aaaiiea.aaea..aeeee
S.STM is adapted from Isum wwonuuuoggww en_ Transformer is adapted from transoormrr_wwee wwwssr_dessseeeeeeeeeeeee _e_en, Dy- namic Cony. is adapted from lightconv iwslt de en.
16

---

Under review as a conference paper at ICLR 2022
Table 7: Data statisties and training time per epoch in seconds. The batch size of each epoch for GEO and Geography is 32, and that for the others is 128.
SCAN
680
ADV
Goosmsptr
Advosiss
Pop
Eig.
Eop.
Eop
Top
Exp
Aug.
5t.
5%.
Dir
Truin Size
09146
205442
20925
20000
30945
724
720
711
728
724
0038
0934
5069
(442
6AB6
915
3814
3600
308340
300240
21300
107614
107014
XSN
19
27
35
Timee TFM
CNN
12
12
12
19
Table 8: Prompts with example primitives and sampled variants. In SCAN, primitives share the same prompt and the number of variants can be changed. In ADV, we randomly sample 5 variants for each source sequence so that we cover all the variants with a test set of an appropriate size.
Data
Primitive
Variant
#Variants
Template
SCAN
jump
jump..
10
[concept] twice
new york city mississippi rivier
houston city
39
how many people in [concept]
red rivier kansas salem
9
how long is [concept] where [concept]
GEO
de dover
49
what states capital [concept]
history of american film aaron magid aaptis 100
advanced ai techniques
5/424
who teaches [concept]
ADV
cargo
5/492
does (concept] give upper-level courses
survmeth
5/1720
core courses for [concept]..
171
5/1895
undergrads toke [concept]
Prompts used in SCAN, GEO, and ADV expressed in Table 8 Detailed experimental results with respect to three levels can be found in Table Table [0] and Table I1
Table 9: Results of Standard inductive learning.
Train
Test
Data
Model
Loss
Token Acc.
Seq. Acc.
Token Acc.
Seq. Acc. %
RNN
0.00 1 0.00
100.00 0.00
99.99 1 0.02
0.00 0.00
99.99 1 0.03
99.95 : 0.08
SCAN
CNN
0.00 + 0.00
99.81 + 0.09
98.78 + 0.55
0.00 0.00
99.96 + 0.08
99.85 0.34
TFM
0.00 = 0.00
99.82 + 0.02
98.83 + 0.12
98.91 + 0.78
97.35 1.62
RNN
0.13 0.02
97.73 + 0.42
80.25 + 2.81
1.36 = 0.48
75.71 + 8.42
44.95 = 14.69
GEO
CNN
0.07 0.01
98.23 + 0.39
76.80 + 2.25
9.01 = 4.26
87.99 + 2.67
69.46 = 5.78
TFM
0.02 0.00
99.63 + 0.07
91.60 + 1.41
4.55 = 1.39
75.37 + 7.84
45.93 = 12.42
RNN
0.03 0.01
99.40 0.13
82.74 + 2.78
6.04 = 0.95
58.61 + 6.18
36.18 5.75
ADV
CNN
0.01 0.01
99.59 1 0.07
85.13 1 1.95
23.56 1 4.95
57. 1 7.55
45.08 1 9.32
TFM
0.00 + 0.00
99.92 + 0.01
96.14 + 0.28
15.12 + 1.00
53.43 + 2.80
42.59 - 3.65
Table 10: Results of Difficult inductive learning.
Train
Test
Model
Loss
Token Acc.
Seq. Acc.
Loss
Token Acc.% %
Seq. Ace. %
RNN
0.00 + 0.00
100.00 0.00
99.99 + 0.01
0.00
99.96 + 0.02
99.85 0.08
SCAN
CNN
0.00 1 0.00
99.77 1 0.19
98.62 1 1.13
99.76 0.54
99.52 1.07
TFM
0.00 1 0.00
99.79 1 0.03
98.59 1 0.12
98.90 1 1.10
96.86 2.64
RNN
0.16 + 0.03
97.39 + 0.67
78.33 + 4.31
1.29 0.27
75.09 + 6.12
43.27 + 13.47
GEO
CNN
0.07 0.01
98.25 + 0.13
76.53 + 1.68
13.87 3.19
79.51 + 6.03
51.20 8.64
TFM
0.00 0.11
99.60 + 0.11
91.33 + 1.46
4.50 0.80
75.11 + 4.88
44.59 9.76
RNN
0.03 + 0.01
99.26 + 0.21
79.57 + 4.12
5.80 + 0.92
59.74 + 5.67
35.69 + 6.05
ADV
CNN
0.02 + 0.00
99.56 + 0.05
$4.06 + 1.57
24.58 - 3.40
54.05 + 5.74
42.14 = 6.90
TFM
0.00 1 0.00
99.91 1 0.01
95.88 1 0.23
15.84 1: 1.51
51.51 1 4.50
41.28 1 4.35
17

---

Under review as a conference paper at ICLR 2022
Table 11: Results of Challenging inductive learning.
Train
Test
Data
Model
Loss
Token Acc. %
Seq. Acc, %
Loss
Token Ace. %
Seq. Acc. %
RNN
0.00 = 0.00
100.00 + 0.00
99.99 + 0.02
0.20 + 0.45
99.95 + 0.08
99.80 + 0.31
SCAN
CNN
0.00 + 0.00
99.85 + 0.05
99.00 + 0.30
0.14 0.31
98.89 + 2.44
97.37 + 5.24
TFM
0.00 1 0.00
99.82 + 0.05
98.85 + 0.27
0.07 + 0.05
98.76 + 0.85
96.38 + 2.81
RNN
0.04
97.76
79.77 +: 4.19
73.46 3.05
36.77 +
GEO
CNN
0.01
98.23 0.17
75.98 + 1.46
15.83 + 4.56
77.40 + 2.48
48.53 3.40
TFM
0.02+0000
99.60 0.06
91.00 1.20
6.01 + 1.03
68.41 + 4.76
36.93 7.47
RNN
0.01 0.03 + 0.01
99.23 + 0.13
79.90 + 1.85
5.95 + 0.90 0.90
58. + 5.82
35.45
6.09
ADV
CNN
0.01 + 0.01
99.68 0.15
87.90 1: 5.05
23.08 + 6.34
53.66 + 2.57
41.37 4.04
TFM
0.00 = 0.00
99.93 0.01
96.41 + 0.24
16.59 - 0.98
49.17 + 2.58
38.88 2.68
Table 12: Concept rules with primitives and their example variants.
Dala
Primilive
Semantic Links
Variant
Concopt Rule
Primitive Rule
Vartant Rule
Jimp look run walk
jumpu)
jump JUMP
jump.p JUMP
SCAN
Lexical Varimet
nm, wall.
run - RUN
1.00K
nin 0 RUN
-LOOK
walk - WALK
waik.0 - WALK
new york city missizziggg
houston
new york city CITY NAME
houston city
CITY NAME
GBO
Co-bypooym
STATE
RIVER NAME
STATE CAPITAL
RIVER NAME
dover
de14s CAPITAL-NAMI
history of american film aaroe magid
advunced di techniques
history of american fim TOPIC auron magid - INSTRUCTOR
advanced ai technigues TOPIC coingo - INSTRUCTOR
ADV
acptis
Co Co-byponym
survwcth 171
cargo
duptls DEPARTMENT
summetth - DEPARTMENT 171 NUMBER
100
100 NUMBER
SEMANTIC LINKING INJECTION VIA DEDUCTIVE LEARNING
Semantic linking can be established via deductive learning, where put concept rules without context information in the training set instead of specific sequence samples. Example concept rules Detailed experimental results with respect to
for SCAN, GEO, and ADV are presented in Table 12]
two levels can be found in Table [3]and and Table 14
Table 13: Results of Standard deductive learning.
Train
Test
Data
Model
Loss
Token Acc. %
Seq. Acc,
Loss
Token Acc. %
Seq. Ace.
RNN
0.00 + 0.00
99.99 + 0.03
99.90 + 0.23
0.05 + 0.06
99.48 + 0.71
98.27 + 2.38
SCAN
CNN
0.00 : 0.00
99.79 + 0.14
98.78 + 0.79
0.00 = 0.00
99.99 + 0.01
99.96 + 0.03
TFM
0.00 1 0.00
99.82 1 0.03
98.78 1 0.17
0.27 1 0.22
96.90 1.78
91.94 1 4.04
RNN
0.17 + 0.03
97.50 + 0.30
78.54 + 2.16
2.83 + 0.69
54.44 + 7.15
13.61 + 7.08
GEO
CNN
0.08 + 0.01
97.97 + 0.24
77.03 + 1.42
51.08 + 25.97
41.86 + 3.38
4.85 + 4.66
TFM
0.02 1 0.00
99.54 1 0.31
91.82 1 2.27
6.03 1 1.56
67.02 1 6.91
36.38 1 10.08
RNN
0.08 + 0.02
98.64 4: 0.31 0.31
68.84 +: 4.57
7.95 1.13
36.50 +: 7.66
12.84 +: 4.31
ADV
CNN
0.02 +: 0.00
99.53 + 0.07
84.64 + 1.20
31. +: 4.76
43.51 + 11.31
32.33 + 12.93
TFM
0.00 + 0.00
99.91 + 0.02
96.33 + 0.37
13.72 + 1.41
36.82 + 3.79
47.43 + 3.71
MACHINE TRANSLATION
We show how semantic linking already benefits models' performance machine translation. The semantic links between primitives and their variants in machine translation is built upon the synony- mous relations between tokens such as "heavily populated" and "populous". Given that synonymous connection is reversible shown in Figure Specifically, we collect a dictionary of tokens for the source language and feed the token to the The
primitive can also be the other primitives' variant.
Google Translation API to obtain token map from the source language to the target same operation can be repeated from the target language to the source one. Two dictionaries are combined into one with duplicates removed. Consequently, we get 144,874 token-level samples as
18

---

Under review as a conference paper at ICLR 2022
Table 14: Results of Difficult deductive learning.
Train
Test
Data
Model
Loss
Token Acc. %
Seq. Ace. %
Loss
Token Acc. %
Seq. Acc. %
RNN
0.00 + 0.00
99.99 + 0.01
99.95 = 0.07
0.08 + 0.08
98.70 + 0.92
95.39 + 2.72
SCAN
CNN
0,00 ::0:::0 0.00
99.62 1 0.34
98.82 = 1.09
0.13 + 0.29
98.50 59 +: 3.10
96.66 + 7.27
TFM
0.00 0.00
99.82 0.03
98.78 0.12
0.21 + 0.20
2.21 96.08
91.26 + 5.80
RNN
0.20 0.03
96.93 1 0.71
75.35 I 3.57
4,40 1 2.50
39.71 1 18.38
7.67 1 5.34
GEO
CNN
0.08 + 0.01
97.77 + 0.76
76.41 2.80
32.94 + 4.26
41.07 + 7.48
4.04 + 2. 18
TFM
0.02 0.00
99.56 +: 0.11
91.08 = 1,56
5.97 + 1.05
65.97 + 5.17
31.57 + 7.42
RNN
0.08 0.02
98.54 1 0.228 0.28
67. L 3.45
7.87 1 1.01
36.42 1. 7.39
12.66 1 5.19
ADV
CNN
0.04 + 0.05
98.78 + 1.91
77.14 + 23.28
32.44 + 6.07
35.34 + 14.68
23.58 + 16.04
TFM
0.00 0.00
99.92 + 0.02
96.41 = 0.26
14.92 = 1.31
53.33 + 3.85
43.24 + 5.14
training supplementary for IWSLT 14 En-De and De-En, and 110,099 for IWSLT En-Fr and Fr- En, which leads to total of 305, 13 training samples for IWSLT' 14 En-De and De-En and 315,671 for IWSLT"15 15 En-Fr and Fr-En after such vocabulary augmentation.
We evaluate our approach on IWSLT' Cetollo (014)) English- German (En-De) and German-English (De-En), IWSLT' 15 Cettolo et al. (2015) English- (En-Fr) and French-English (Fr-En) translation tasks. We follow the standard evaluation protocol Ott et al. (019) that keeps the original training set and validation set but combines multiple previous test sets for final evaluation We apply BPE with 10K tokens for all tasks and report both BLEU Pa- pineni et al. (2002) and SacreBLEU Post (2018) scores for three baselines: LSTM Luong et al. (2015). Transformer Vaswani et al. (2017). and Dynamic Conv. Wu et al. (2019) in comparsion with same structures augmented by our method.
Experimental Setup.
we observe a consistent improvement in both BLEU and SacreBLEU over all baselines when performed vocabulary augmentation, particularly up to 1 in SacreBIEU. The ad- ditional synonym pairs not only construct the semantic linking between tokens in two languages explicitly, but also create a complicated semantic linking network implicitly because of synonyms within the single language and the transitivity nature of synonym relation. Our experiments prove that semantic linking, which allows models generalize systematically, is also beneficial for im- proving machine translation performance.
Results. From Table
SEMANTIC PARSING
We consider a variable as a hypernym for its values and entities belonging to the same variable as co-hyponyms in semantic parsing. Thus, we can treat all the entity values for all variables as primitives and the translations from primitives to their variables as primitive rules that later joins the base training set. For fair comparison, a token from this extra dataset will be marked as a unique unknown mark, "junk;", if it does not exist in the original base training set. After that, we have a map of 103 entity translations for Geography and 1846 for Advising, resulting in a training size change from 701 to 804 for Geography and from 3814 to 5660 for Advising.
Experimental Setup. We evaluate our method on two semantic parsing benchmarks, Geography, and Advising. We train the same models as we analyzed before without more hyperparameter tuning, including RNN, CNN, and TFM. There are some changes for CNN, where the learning rate is Se -4 in Geography, and the maximum sequence length for the decoder position embedding is 312 in Advising. We split 10% training samples as the validation set to find the converged epoch and then add it back to the training set for the final report.
As claborated in Table 3 all three networks can achieve better performance in terms of both accuracy and variance. Specifically, a 10.76% token accuracy and 9.95% sequence accuracy boosting are observed from RNN on Advising after such entity augmentation. The results suggest that models can learn semantic linking be more familiar with similar contexts from those primi- tive rules in deductive way to enhance model systematic generalization and finally lead to better outcomes.
Results.
9The final test set of IWSLT' 2011, 2012) and TWSLT'15 15 includes W.S.TI..TE...sttf....... 2015) Ont al. (2019).
14 consists of IWSL.14.TTD.. dev seddddv(220) (2010, 2012] and IWSI.114.TDD.ss/000
19


Reviews:
[{"review_id": "k8Jp0dCNC5I", "reviewer": "Reviewer_yQN2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Merits:\n* This paper focuses on an important problem - systematic genneralization and compositional generalization in particular.\n* The experimental evaluation is sound including both synthetic and NMT tasks. The evaluation protocal and the design of three tasks are valuable for followups.\n* The empirical performances are encouraging and interesting.\n\nFlaws:\n* The model depends heavily on the prior knowledge about the structure of the task but essentially a more realistic approach towards compositionality is concerning how to induce gramatical rules over raw natural language (considering the birttleness of many neural-symbolic systems).\n* Considering there are already many works using data augmentation to improved compositional generalizaiton [1,2], the algorithmic novelty only combining with semantic linking is quite doubleful in my opinion. It is interesting to unify existing (data-augmentation) frameworks but the efforts are not solid enough.\n* The inductive and deductive processes are not very well justified. For example, in symbolism, inductive learning induce (logical) rules explicitly by composing predicates. But in the paper, it seems the model only use rules to generate extra parrallel data for training.\n* It would be nice to discuss more on the connections between the intruiging performance in synthetic dataset like SCAN and the systematic generalization capability over real-world cases [3, 4]. Considering there are many works on compositional \nwhether they are only exploiting dataset biases is an open question and suits the title of this work.\n* The claims in the paper are a bit vague like 'meaningful learning' (seems like a cognitive concept but not very clear to many authors). And the experiment part is well-written but the previous parts are not as good as it. It would be much better to have an algorithm for introducing the method.\n\n[1] Learning to Recombine and Resample Data for Compositional Generalization\n\n[2] Good-Enough Compositional Data Augmentation\n\n[3] The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers\n\n[4] Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN\n", "summary_of_the_paper": "This paperintroduce semantic linking for systematic generalization through the analysis of inductive and deductive learning from a meaningful learning perspective. They show that both prior knowledge and semantic linking play a key role in\nsystematic generalization, which is in line with the so-called 'meaningful learning theory'. Interesting results are attained from\nSCAN to real data.", "summary_of_the_review": "The paper has some merits but the overall claims are not well justified. The work can be improved after a major revision and narrow down the claim a bit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dCX5kICXxHp", "reviewer": "Reviewer_NJJw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think the presentation of the algorithm can be further improved by walking through a concrete example, especially for 2.2 and 2.3. For example, when the authors says, \"getting prompt by replacing words with slot mark\", it will be great if we can get a concrete example showing what's exactly the prompt. Meanwhile, the arrows in Figure 1 is kind of confusing. For example, why is the arrow from \"prior knowledge\" to the bottom middle box the same type of arrow as the one from the bottom middle box to \"new compositions?\"\n\nMy primary concerns about this paper are the following.\n\n1. The learning setup contains very strong prior knowledge about the domain, which, sometimes can be wrong. For example, in real-world datasets because of pragmatics, \"red\" and \"green\" might not be always interchangeable: \"you have a green light.\"\n2. If we view the algorithm just as a data augmentation technique, I believe, at least for the cases studied in this paper, it has been covered by many existing techniques:\n- https://arxiv.org/abs/2011.09039 This paper contains results for a number of heuristic augmentation techniques.\n- https://arxiv.org/abs/1904.09545 \n- https://arxiv.org/abs/2010.03706 Learning to do data augmentation.\n- https://aclanthology.org/2021.findings-acl.307.pdf (replacing a subtree with a new tree that has the same tag)\n\nThe authors discussed a bit about some related works in the second paragraph in Section 4, but I am still not seeing enough contribution compared with existing methods. For example, can I interpret the authors' comments on existing replacement-based augmentation as:\n\nExisting methods use automatic ways to find possible replacements to augment the data (e.g., from a few occurrences, trying to guess that jump_0 and jump can be used interchangeably), however, if we know more about the domain, we can just write Python programs to tell the model that jump_0 and jump can be used interchangeably (this paper)?\n\nIf I'm going to be harsh, I would like to argue that in the settings studied in this paper (See Table 12 in the appendix), all these tasks can be performed simply by replacing \"jump_0\" with \"jump\" (\"Houston city\" with \"New York city\", etc) during test time. This assumes the same kind of knowledge as the authors' proposed technique.\n\n3. I am not sure about the significance of having multiple splits in Table 2. Why is this setting meaningful? If we already know the heuristic rules, then what's the scenario where we need to exclude primitive training samples of the same length as their variant samples...?\nNote that these generalization tests are all \"easier\" than the one we have in SCAN (e.g., jump generalization, where there is only one single example of the novel word \"jump\")\n", "summary_of_the_paper": "This paper considers the problem of learning novel words from a few examples. The authors name their approach as \"meaningful learning,\" which, at a high level, means that we should relate the new word with existing words. The concrete technique they proposed is to use domain-specific rules to generate new data that contains novel words based on the existing examples.", "summary_of_the_review": "The paper is decently written and the statements are well supported. However, I failed to see the significance of the paper nor its contribution compared with other existing papers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EfO_lUcMW-G", "reviewer": "Reviewer_PBxU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The interpretation of system generalization from the psychological concepts of meaningful learning and semantic linking is really interesting and reasonable. \n2. The author designed extensive experiments to prove the concepts and also show the effectiveness on real data - machine translation and semantic parsing.\n\nWeaknesses:\n1. Although the author is well motivated by some psychological concepts such as meaningful learning, semantic linking or prior knowledge, I had a hard time to draw the connection between these concepts to targeted systematic generation tasks (e.g., SCAN). I think the author should give a more concise definition of these concepts, and improve the examples in Figure 1 with better explanation.\n2. Although the perspective of inductive and deductive learning sounds fancy and reasonable, the introduced data augmentation methods are quite simple, and a bit frustrating. \n3. It seems this paper only focuses on a specific type of generalization - lexical variants, while most of the systematic generation research focuses on compositionality. The paper seems to overclaim and not be clear at the early part of the paper.", "summary_of_the_paper": "This paper revisits the problem of neural network's systematic generalization ability from the perspective of meaningful learning, or more specifically, semantic linking. Based on this view, they propose two data augmentation methods from either the inductive learning perspective or deductive learning perspective. They train different model variants with such augmented data. The empirical results on SCAN, GEO, and ADV show that models can behave systematically. They further group some data augmentation methods on the machine translation task and semantic parsing task into the inductive or the deductive category, and show these augmentation methods can bring benefit in real data.\n", "summary_of_the_review": "This paper provides a new perspective of systematic generalization, and gives empirical results to prove that methods based on this perspective can improve model's generalization on toy and real tasks. However, the explanation of the new perspective is not clear enough, and the proposed methods look pretty simple and only solve simple generalization problems.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ee6-HCw2UUr", "reviewer": "Reviewer_LcKh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The data augmentation method is interesting, bringing semantic linking to bridge the new and old concepts. Deductive learning and inductive learning provide two learning directions for specific-to-general and general-to-specific.\n\n2. It designs and conducts comprehensive experiments for analyzing how the semantic linking in inductive learning and deductive learning affects the models. The ablation study shows how various primitives impact the model.\n\nWeaknesses: \n\n1. The technical contribution is weak. It proposes the augmentation approach using the semantic link but does not study how to better use the semantic link. I would like to see more insights into how to design the model beyond these traditional sequence-to-sequence models.\n\n2. It talks a lot about inductive learning and deductive learning but does not provide learning approaches in the deductive style.\n\n3. Some arguments and claims are not appropriate and make it harder to understand the approaches. For example, the two methods for injecting the semantic link are actually rules with or without context. The inductive learning and deductive learning story make it ambiguous on whether it is a learning strategy or data augmentation strategy. I suggest the proper tone should be augmenting the data and then explaining the learning process with augmented data in two paradigms, but not listing the inductive/deductive learning at the beginning, which makes it very confusing.\n", "summary_of_the_paper": "The paper introduces an interesting idea of improving the systematic generalization ability via meaningful learning. Through providing augmented data for inductive learning and deductive learning, the sequence-to-sequence model can be more generalizable to compositions of new concepts. It tests on real data to provide evidence of the efficacy. ", "summary_of_the_review": "The technical contribution is relatively weak and the arguments and stories of this paper make it hard to understand the specific contribution of each part. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to rationalize data augmentation techniques for compositional generalization by evoking the principle of meaningful
learning which posits that learning new concepts builds on previously learned concepts (which learners already understand). So for compositional generalization, this means that a model exposed to some new concepts in the test set, should link them to known concepts which have been already attested in the training set. The links between concepts are presumed to be semantic, e.g., hyponyms, hypernyms, or lexical variants. Ideally, a model should perform semantic linking on its own, however the authors do not propose a linking mechanism. Rather they investigate data augmentation as a way of exposing a model to semantic links and then explore whether different operationalizations of semantic linking enable the model to generalize better. Inductive learning is a bottom-up approach, where links are created from general to specific concepts, whereas deductive learning is a top-down approach where links are created from specific to general concepts. Experimental results indicate that inductive learning works better. 

The reviewers had the following issues with the submission (a) the technical contribution is not very strong (the idea of data augmentation is not new, although the authors' meaningful learning perspective is) (b) semantic linking seems to be able to handle only cases pertaining to lexical generalization (even though the authors include examples with structural generalization in their splits, there is no reason why semantic linking could handle these cases); (c) it would be more interesting/useful  to learn the linking than assume it is given. The authors did their best to respond to the criticism, but ultimately addressing the criticism is future work. I would also recommend to take a look at this dataset which might be useful for machine learning experiments: https://arxiv.org/abs/2105.14802
Decision: Reject