From SCAN to Real Data: Systematic Generalization via Meaningful Learning

1 INTRODUCTION . As a crucial characteristic of human cognition , systematic generalization reflects people ’ s ability to learn infinite combinations of finite concepts ( Chomsky , 1957 ; Montague et al. , 1970 ) . However , weak systematic compositionality has been considered as a primary obstacle to the expression of language and thought in connectionist networks for a long time ( Fodor & Pylyshyn , 1988 ; Hadley , 1994 ; Marcus , 1998 ; Fodor & Lepore , 2002 ; Frank et al. , 2009 ; Brakel & Frank , 2009 ; Marcus , 2018 ) . Whether models can generalize systematically is still an appealing research topic until now . Recent works state that modern neural networks have not mastered these language-based generalization challenges in multiple explicitly proposed datasets ( Lake & Baroni , 2017 ; Bastings et al. , 2018 ; Keysers et al. , 2019 ; Hupkes et al. , 2020 ; Kim & Linzen , 2020 ) . These studies conclude that models lack such cognitive capacity , which calls for a more systematic study . Apart from the proposal of benchmarks , existing research mainly focuses on novel architectural designs ( Chen et al. , 2020 ) data augmentation ( Andreas , 2020 ; Akyürek et al. , 2021 ) and meta-learning ( Lake , 2019 ; Conklin et al. , 2021 ) to enable systematic generalization . In this work , however , we question that whether neural networks are indeed deficient or just conventional learning protocols unable to exploit their full potential ( Csordás et al. , 2021 ) . Inspired by meaningful learning from the field of educational psychology ( Mayer , 2002 ) , we revisit systematic generalization to see whether neural networks still fail after semantic linking . To exposure semantic links between new concepts and existing ones , we augment prior knowledge through either inductive learning or deductive learning as what humans do in meaningful verbal learning ( Ausubel , 1963 ) . To be specific , inductive learning is a bottom-up approach from the more specific to the mode general . By introducing new concepts sharing the same context with existing ones in specific samples , we hope the model can capture the underlying semantic connections and thus generalize to novel compositions of new concepts . On the contrary , deductive learning is a top-down approach from the more general to the more specific . By involving a rule-like concept dictionary without specific context information , we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking . In experiments , inductive and deductive learning stand for training models on extra samples with or without context , respectively , as two standard data augmentation techniques ( Arthur et al. , 2016 ; Wei & Zou , 2019 ; Nag et al. , 2020 ) . We mainly focus on three semantic relationships , namely , lexical variant , co-hyponym , and synonym . Starting from SCAN , our experiments confirm that , with semantic linking , even canonical neural networks can generalize systematically to new concepts and compositions . Moreover , this holds consistent across two more semantic parsing datasets . As an ablation study , we further examine such one-shot compositional generalization and find that both prior knowledge and semantic linking take essential parts . Lastly , we extend from toy sets to real data and explain how models ’ meaningful learning capability benefits them in solving real problems such as machine translation and semantic parsing with the assistance of data augmentation techniques . Overall , our contributions are as follows : ( 1 ) We formally revisit systematic generalization by introducing semantic linking from a meaningful learning perspective . ( 2 ) We show how to conduct semantic linking by two common data augmentation approaches , either inductively or deductively . ( 3 ) We observe that modern neural networks can achieve systematic generalization with semantic linking , and both prior knowledge and semantic linking play a key role , which is in line with meaningful learning theory . ( 4 ) We extend from SCAN to real data and demonstrate that many recent data augmentation techniques belong to either inductive or deductive learning . 2 MEANINGFUL LEARNING . Learning new concepts by relating them to the existing ones is defined as a process of meaningful learning in educational psychology ( Ausubel , 1963 ; Mayer , 2002 ) . The utilization of meaningful learning can encourage learners to understand information continuously built on concepts the learners already understand ( Okebukola & Jegede , 1988 ) . Following the same idea , we intend to examine models ’ systematic compositionality by exploring semantic linking that establishes semantic relations between primitives P ( old concepts ) and their variants V : = { Vp | ∀p ∈ P } ( new concepts ) . To spoon-feed semantic knowledge to models for semantic linking , we augment the training data by either inductive learning or deductive learning ( Hammerly , 1975 ; Shaffer , 1989 ; Thornbury , 1999 ) as humans learning vocabulary . In this section , we discuss the process of semantic linking and take “ jump ” from SCAN as an example primitive to illustrate the learning scheme in Figure 1 . 2.1 SEMANTIC LINKS . We aim to revisit systematic generalization by exposing semantic links such as lexical variants , cohyponyms , and synonyms . Lexical Variant refers to an alternative expression form for the same concept . Co-hyponym is a linguistic term to designate a semantic relation between two group members belonging to the same broader class , where each member is a hyponym and the class is a hypernym ( Lyons & John , 1995 ) . Synonym stands for a word , morpheme , or phrase that shares exactly or nearly the same semantics with another one . We provide an example and a detailed description in Appendix . 2.2 INDUCTIVE LEARNING . Inductive learning is a bottom-up approach from the more specific to the more general . In grammar teaching , inductive learning is a rule-discovery approach that starts with the presentation of specific examples from which a general rule can be inferred ( Thornbury , 1999 ) . Inspired by that , we propose to conduct semantic linking by introducing variant samples sharing the same context with their primitives during training . The assumption is that models can observe the interchange of primitives and their variants surrounded by the same context in the hope of coming up with a general hypothesis that there is a semantic linking between primitives and their variants ( Harris , 1954 ) . Formally , we describe inductive learning as follows . For a sequence-to-sequence task T : X → Y , we have a source sequence x ∈ X and its target sequence y ∈ Y . We prepare prompts set Z : = { z = fprompt ( x ) | x ∈X } , where fprompt ( · ) replaces the primitive in x with a slot mark [ zp ] .1 Then , we generate XIL : = { xIL = ffill ( z , v ) | z ∈ Z , v ∈ V } by filling [ zp ] with variants in Vp . There is no change from the target side , so we get Y IL by copying y as yIL for each xIL correspondingly . Finally , we train models on ( [ X XIL ] , [ Y Y IL ] ) to operate semantic linking inductively . In practice , as shown in Figure 1 , we first get a prompt “ [ Zjump ] twice ” given a primitive sample “ jump twice ” . After that , we generate variant samples by replacing the slot mask “ [ Zjump ] ” with variants such as “ jump 0 ” . Finally , training models on generated variant samples like “ jump 0 twice ” combined with prior knowledge , we aim to establish the semantic relationships between primitives and their variants inductively . This process can be also treated as a kind of replacement augmentation ( Wei & Zou , 2019 ) . 2.3 DEDUCTIVE LEARNING . Deductive Learning , on the opposite of inductive learning , is a top-down approach from the more general to the more specific . As a rule-driven approach , teaching in a deductive manner often begins with presenting a general rule and is followed by specific examples in practice where the rule is applied ( Thornbury , 1999 ) . To align with this definition , we intend to do semantic linking deductively by combining a bilingual dictionary that maps primitives and their variants to the same in the target domain . This additional dictionary , hence , mixes the original training task with word translation ( Mikolov et al. , 2013b ) . Without any specific context , we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking . Formally , we describe deductive learning as follows . We first treat P as the source dataset XDLP directly and then prepare the corresponding target dataset Y DLP by either decomposing samples from Y manually or feeding XDLP to a trained external model . Similarly , we can consider V as another source dataset XDLV and prepare its target dataset Y DL V by copying the corresponding y DL P as y DL V for all x DL V as variants of each xDLP . After all , we get X DL as [ XDLP XDLV ] and Y DL as [ Y DLP Y DLV ] . The mapping from XDL to Y DL is a dictionary to translate primitives and their variants to the same targets without any specific context information . We name ( xDL , yDL ) as a concept rule , ( xDLP , y DL P ) as a primitive rule , and ( xDLV , y DL V ) as a variant rule since they are more rule-like without contexts . We train models on ( [ X XDL ] , [ Y Y DL ] ) to operate semantic linking deductively . In practice , as presented in Figure 1 , we directly make use of primitive “ jump ” and its variants such as “ jump 0 ” as the source sequences with action “ JUMP ” as their same target sequences . By exposing both the primitive rule “ jump ” → “ JUMP ” and variants rules like “ jump 0 ” → “ JUMP ” during training , we intend to build the semantic connections between primitives and their variants deductively . This manner is similar to the bilingual dictionary augmentation ( Arthur et al. , 2016 ; Nag et al. , 2020 ) . 3 SYSTEMATIC GENERALIZATION . Although previous studies argue that neural networks fail to match humans in systematic generalization ( Lake & Baroni , 2017 ; Keysers et al. , 2019 ) , we revisit such algebraic compositionality conditioned on the semantic linking to see whether the conclusion will change . The following section moves on to specify the process and outcome of experiments . We first intend to make use of SCAN as the initial testbed to observe the presence of systematic generalization with the assistance of semantic relations . Then , we verify neural networks ’ potential to achieve the systematic generalization activated by semantic linking on SCAN , as well as two real-world tasks of semantic parsing . Following ablation studies further examine models ’ compositional capability . 1For each primitive , we pick only one prompt to fill in all its variants with Z specified for various datasets in Appendix .

Summary: The paper introduces an interesting idea of improving the systematic generalization ability via meaningful learning. Through providing augmented data for inductive learning and deductive learning, the sequence-to-sequence model can be more generalizable to compositions of new concepts. It tests on real data to provide evidence of the efficacy.