From SCAN to Real Data: Systematic Generalization via Meaningful Learning

1 INTRODUCTION . As a crucial characteristic of human cognition , systematic generalization reflects people ’ s ability to learn infinite combinations of finite concepts ( Chomsky , 1957 ; Montague et al. , 1970 ) . However , weak systematic compositionality has been considered as a primary obstacle to the expression of language and thought in connectionist networks for a long time ( Fodor & Pylyshyn , 1988 ; Hadley , 1994 ; Marcus , 1998 ; Fodor & Lepore , 2002 ; Frank et al. , 2009 ; Brakel & Frank , 2009 ; Marcus , 2018 ) . Whether models can generalize systematically is still an appealing research topic until now . Recent works state that modern neural networks have not mastered these language-based generalization challenges in multiple explicitly proposed datasets ( Lake & Baroni , 2017 ; Bastings et al. , 2018 ; Keysers et al. , 2019 ; Hupkes et al. , 2020 ; Kim & Linzen , 2020 ) . These studies conclude that models lack such cognitive capacity , which calls for a more systematic study . Apart from the proposal of benchmarks , existing research mainly focuses on novel architectural designs ( Chen et al. , 2020 ) data augmentation ( Andreas , 2020 ; Akyürek et al. , 2021 ) and meta-learning ( Lake , 2019 ; Conklin et al. , 2021 ) to enable systematic generalization . In this work , however , we question that whether neural networks are indeed deficient or just conventional learning protocols unable to exploit their full potential ( Csordás et al. , 2021 ) . Inspired by meaningful learning from the field of educational psychology ( Mayer , 2002 ) , we revisit systematic generalization to see whether neural networks still fail after semantic linking . To exposure semantic links between new concepts and existing ones , we augment prior knowledge through either inductive learning or deductive learning as what humans do in meaningful verbal learning ( Ausubel , 1963 ) . To be specific , inductive learning is a bottom-up approach from the more specific to the mode general . By introducing new concepts sharing the same context with existing ones in specific samples , we hope the model can capture the underlying semantic connections and thus generalize to novel compositions of new concepts . On the contrary , deductive learning is a top-down approach from the more general to the more specific . By involving a rule-like concept dictionary without specific context information , we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking . In experiments , inductive and deductive learning stand for training models on extra samples with or without context , respectively , as two standard data augmentation techniques ( Arthur et al. , 2016 ; Wei & Zou , 2019 ; Nag et al. , 2020 ) . We mainly focus on three semantic relationships , namely , lexical variant , co-hyponym , and synonym . Starting from SCAN , our experiments confirm that , with semantic linking , even canonical neural networks can generalize systematically to new concepts and compositions . Moreover , this holds consistent across two more semantic parsing datasets . As an ablation study , we further examine such one-shot compositional generalization and find that both prior knowledge and semantic linking take essential parts . Lastly , we extend from toy sets to real data and explain how models ’ meaningful learning capability benefits them in solving real problems such as machine translation and semantic parsing with the assistance of data augmentation techniques . Overall , our contributions are as follows : ( 1 ) We formally revisit systematic generalization by introducing semantic linking from a meaningful learning perspective . ( 2 ) We show how to conduct semantic linking by two common data augmentation approaches , either inductively or deductively . ( 3 ) We observe that modern neural networks can achieve systematic generalization with semantic linking , and both prior knowledge and semantic linking play a key role , which is in line with meaningful learning theory . ( 4 ) We extend from SCAN to real data and demonstrate that many recent data augmentation techniques belong to either inductive or deductive learning . 2 MEANINGFUL LEARNING . Learning new concepts by relating them to the existing ones is defined as a process of meaningful learning in educational psychology ( Ausubel , 1963 ; Mayer , 2002 ) . The utilization of meaningful learning can encourage learners to understand information continuously built on concepts the learners already understand ( Okebukola & Jegede , 1988 ) . Following the same idea , we intend to examine models ’ systematic compositionality by exploring semantic linking that establishes semantic relations between primitives P ( old concepts ) and their variants V : = { Vp | ∀p ∈ P } ( new concepts ) . To spoon-feed semantic knowledge to models for semantic linking , we augment the training data by either inductive learning or deductive learning ( Hammerly , 1975 ; Shaffer , 1989 ; Thornbury , 1999 ) as humans learning vocabulary . In this section , we discuss the process of semantic linking and take “ jump ” from SCAN as an example primitive to illustrate the learning scheme in Figure 1 . 2.1 SEMANTIC LINKS . We aim to revisit systematic generalization by exposing semantic links such as lexical variants , cohyponyms , and synonyms . Lexical Variant refers to an alternative expression form for the same concept . Co-hyponym is a linguistic term to designate a semantic relation between two group members belonging to the same broader class , where each member is a hyponym and the class is a hypernym ( Lyons & John , 1995 ) . Synonym stands for a word , morpheme , or phrase that shares exactly or nearly the same semantics with another one . We provide an example and a detailed description in Appendix . 2.2 INDUCTIVE LEARNING . Inductive learning is a bottom-up approach from the more specific to the more general . In grammar teaching , inductive learning is a rule-discovery approach that starts with the presentation of specific examples from which a general rule can be inferred ( Thornbury , 1999 ) . Inspired by that , we propose to conduct semantic linking by introducing variant samples sharing the same context with their primitives during training . The assumption is that models can observe the interchange of primitives and their variants surrounded by the same context in the hope of coming up with a general hypothesis that there is a semantic linking between primitives and their variants ( Harris , 1954 ) . Formally , we describe inductive learning as follows . For a sequence-to-sequence task T : X → Y , we have a source sequence x ∈ X and its target sequence y ∈ Y . We prepare prompts set Z : = { z = fprompt ( x ) | x ∈X } , where fprompt ( · ) replaces the primitive in x with a slot mark [ zp ] .1 Then , we generate XIL : = { xIL = ffill ( z , v ) | z ∈ Z , v ∈ V } by filling [ zp ] with variants in Vp . There is no change from the target side , so we get Y IL by copying y as yIL for each xIL correspondingly . Finally , we train models on ( [ X XIL ] , [ Y Y IL ] ) to operate semantic linking inductively . In practice , as shown in Figure 1 , we first get a prompt “ [ Zjump ] twice ” given a primitive sample “ jump twice ” . After that , we generate variant samples by replacing the slot mask “ [ Zjump ] ” with variants such as “ jump 0 ” . Finally , training models on generated variant samples like “ jump 0 twice ” combined with prior knowledge , we aim to establish the semantic relationships between primitives and their variants inductively . This process can be also treated as a kind of replacement augmentation ( Wei & Zou , 2019 ) . 2.3 DEDUCTIVE LEARNING . Deductive Learning , on the opposite of inductive learning , is a top-down approach from the more general to the more specific . As a rule-driven approach , teaching in a deductive manner often begins with presenting a general rule and is followed by specific examples in practice where the rule is applied ( Thornbury , 1999 ) . To align with this definition , we intend to do semantic linking deductively by combining a bilingual dictionary that maps primitives and their variants to the same in the target domain . This additional dictionary , hence , mixes the original training task with word translation ( Mikolov et al. , 2013b ) . Without any specific context , we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking . Formally , we describe deductive learning as follows . We first treat P as the source dataset XDLP directly and then prepare the corresponding target dataset Y DLP by either decomposing samples from Y manually or feeding XDLP to a trained external model . Similarly , we can consider V as another source dataset XDLV and prepare its target dataset Y DL V by copying the corresponding y DL P as y DL V for all x DL V as variants of each xDLP . After all , we get X DL as [ XDLP XDLV ] and Y DL as [ Y DLP Y DLV ] . The mapping from XDL to Y DL is a dictionary to translate primitives and their variants to the same targets without any specific context information . We name ( xDL , yDL ) as a concept rule , ( xDLP , y DL P ) as a primitive rule , and ( xDLV , y DL V ) as a variant rule since they are more rule-like without contexts . We train models on ( [ X XDL ] , [ Y Y DL ] ) to operate semantic linking deductively . In practice , as presented in Figure 1 , we directly make use of primitive “ jump ” and its variants such as “ jump 0 ” as the source sequences with action “ JUMP ” as their same target sequences . By exposing both the primitive rule “ jump ” → “ JUMP ” and variants rules like “ jump 0 ” → “ JUMP ” during training , we intend to build the semantic connections between primitives and their variants deductively . This manner is similar to the bilingual dictionary augmentation ( Arthur et al. , 2016 ; Nag et al. , 2020 ) . 3 SYSTEMATIC GENERALIZATION . Although previous studies argue that neural networks fail to match humans in systematic generalization ( Lake & Baroni , 2017 ; Keysers et al. , 2019 ) , we revisit such algebraic compositionality conditioned on the semantic linking to see whether the conclusion will change . The following section moves on to specify the process and outcome of experiments . We first intend to make use of SCAN as the initial testbed to observe the presence of systematic generalization with the assistance of semantic relations . Then , we verify neural networks ’ potential to achieve the systematic generalization activated by semantic linking on SCAN , as well as two real-world tasks of semantic parsing . Following ablation studies further examine models ’ compositional capability . 1For each primitive , we pick only one prompt to fill in all its variants with Z specified for various datasets in Appendix .

Summary: This paper revisits the problem of neural network's systematic generalization ability from the perspective of meaningful learning, or more specifically, semantic linking. Based on this view, they propose two data augmentation methods from either the inductive learning perspective or deductive learning perspective. They train different model variants with such augmented data. The empirical results on SCAN, GEO, and ADV show that models can behave systematically. They further group some data augmentation methods on the machine translation task and semantic parsing task into the inductive or the deductive category, and show these augmentation methods can bring benefit in real data.