Cognition 164 (2017) 116–143

Contents lists available at ScienceDirect

Cognition
journal homepage: www.elsevier.com/locate/COGNITBootstrapping language acquisition q
Omri Abend ⇑,1, Tom Kwiatkowski 2, Nathaniel J. Smith 3, Sharon Goldwater, Mark Steedman
Informatics, University of Edinburgh, United Kingdom

a r t i c l e

i n f oArticle history:
Received 10 February 2016
Revised 1 November 2016
Accepted 22 February 2017

Parts of this work were previously
presented at EMNLP 2010, EACL 2012, andappeared in Kwiatkowski’s doctoral
dissertation
Keywords:
Language acquisition
Syntactic bootstrapping
Semantic bootstrapping
Computational modeling
Bayesian model
Cross-situational learninga b s t r a c t
The semantic bootstrapping hypothesis proposes that children acquire their native language throughexposure to sentences of the language paired with structured representations of their meaning, whose
component substructures can be associated with words and syntactic structures used to express theseconcepts. The child’s task is then to learn a language-specific grammar and lexicon based on (probably
contextually ambiguous, possibly somewhat noisy) pairs of sentences and their meaning representations(logical forms).
Starting from these assumptions, we develop a Bayesian probabilistic account of semantically bootstrapped first-language acquisition in the child, based on techniques from computational parsing andinterpretation of unrestricted text. Our learner jointly models (a) word learning: the mapping between
components of the given sentential meaning and lexical words (or phrases) of the language, and (b) syntax learning: the projection of lexical elements onto sentences by universal construction-free syntactic
rules. Using an incremental learning algorithm, we apply the model to a dataset of real syntactically complex child-directed utterances and (pseudo) logical forms, the latter including contextually plausible but
irrelevant distractors. Taking the Eve section of the CHILDES corpus as input, the model simulates several
well-documented phenomena from the developmental literature. In particular, the model exhibits syntactic bootstrapping effects (in which previously learned constructions facilitate the learning of novel
words), sudden jumps in learning without explicit parameter setting, acceleration of word-learning(the ‘‘vocabulary spurt”), an initial bias favoring the learning of nouns over verbs, and one-shot learningof words and their meanings. The learner thus demonstrates how statistical learning over structured representations can provide a unified account for these seemingly disparate phenomena.
Ó 2017 Elsevier B.V. All rights reserved.

1. Introduction
One of the fundamental challenges facing a child language learner is the problem of generalizing beyond the input. Using various
social and other extralinguistic cues, a child may be able to workout the meaning of particular utterances they hear, like ‘‘you readthe book” or ‘‘Eve will read Lassie”, if these are encountered in theappropriate contexts. But merely memorizing and reproducing
earlier utterances is not enough: children must also somehowq
The work was supported in part by ERC Advanced Fellowship 249520
GRAMPLUS, EU IST Cognitive Systems IP EC-FP7-270273 Xperience, ARC Discovery
grants DP 110102506 and 160102156, and a James S McDonnell Foundation ScholarAward.
⇑ Corresponding author at: School of Computer Science and Engineering, Rothberg Building A527, Edmond Safra Campus, Givat Ram, The Hebrew University,
Jerusalem 91904, Israel.
E-mail address: oabend@cs.huji.ac.il (O. Abend).
1
Now at the Departments of Computer Science & Cognitive Science, The Hebrew
University of Jerusalem.
2
Now at Google Research.
3
Now at the Berkeley Institute of Data Science, University of California, Berkeley.

http://dx.doi.org/10.1016/j.cognition.2017.02.009
0010-0277/Ó 2017 Elsevier B.V. All rights reserved.

use these experiences to learn to produce and interpret novel
utterances, like ‘‘you read Lassie” and ‘‘show me the book”. There
are many proposals for how this might be achieved, but abstractly
speaking it seems to require the ability to explicitly or implicitly (a)decompose the utterance’s form into syntactic units, (b) decompose the utterance’s meaning into semantic units, (c) learn lexical
mappings between these syntactic and semantic units, and (d)learn the language-specific patterns that guide their recombination(so that e.g. ‘‘Eve will read Lassie to Fraser”, ‘‘will Eve read Fraser
Lassie?”, and ‘‘will Fraser read Eve Lassie?” have different meanings,
despite using the same or nearly the same words). A further challenge is that even in child-directed speech, many sentences are
more complex than ‘‘you read Lassie”; the child’s input consistsof a mixture of high- and low-frequency words falling into a variety of syntactic categories and arranged into a variety of more orless complex syntactic constructions.
In this work, we present a Bayesian language-learning model
focused on the acquisition of compositional syntax and semantics
in an incremental, naturalistic setting. That is, our model receives
training examples consisting of whole utterances paired with noisy

O. Abend et al. / Cognition 164 (2017) 116–143

representations of the whole utterance’s meaning, and from theseit learns probabilistic representations of the semantics and syntax
of individual words, in such a way that it becomes able to recombine these words to understand novel utterances and express novel
meanings. This requires that the model simultaneously learn how
to parse syntactic constructions, assign meaning to specific words,
and use syntactic regularities (for example, in verb argument
structure) to guide interpretation of ambiguous input. Our training
data consists of real, syntactically complex child-directed utterances drawn from a single child in the CHILDES corpus, and ourtraining is incremental in the sense that the model is presented
with each utterance exactly once, in the same order that the child
actually encountered them.
The work described here represents an advance over previous
models that focused on learning either word meanings or syntax
given the other (see below for a review). By developing a joint
learning model we are able to explore how these phenomena interact during learning. A handful of other joint learning models havebeen presented in the literature, but these have either worked fromsynthetic input with varying degrees of realism (Beekhuizen, 2015;
Maurits, Perfors, & Navarro, 2009) or have not yet been evaluatedon specific phenomena known from child language acquisition,
as we do here (Chrupała, Kádár, & Alishahi, 2015; Jones, 2015). In
particular, we show in a series of simulations that our model exhibits syntactic bootstrapping effects (in which previously learned
constructions facilitate the learning of novel words), sudden jumps
in learning without explicit parameter setting, acceleration ofword-learning (the ‘‘vocabulary spurt”), an initial bias favoring
the learning of nouns over verbs, and one-shot learning of words
and their meanings. These results suggest that there is no needto postulate distinct learning mechanisms to explain thesevarious phenomena; rather they can all be explained through a single mechanism of statistical learning over structured
representations.
1.1. Theoretical underpinnings
Our model falls under the general umbrella of ‘‘Semantic Bootstrapping” theory, which assumes that the child can access a structural representation of the intended semantics or conceptual
content of the utterance, and that such representations are sufficiently homomorphic to the syntax of the adult language for a
mapping from sentences to meanings to be determined(Bowerman, 1973; Brown, 1973; Clark, 1973; Grimshaw, 1981;
Pinker, 1979; Schlesinger, 1971; cf. Wexler & Culicover, 1980:78–
84; Berwick, 1985:22–24). By ‘‘homomorphic”, we simply meanthat meaning representation and syntax stand in a ‘‘type-to-type”
relation, according to which every syntactic type (such as the English intransitive verb) corresponds to a semantic type (such as thepredicate), and every rule (such as English S ! NP VP) corresponds to a semantic operation (such as function application of
the predicate to the subject).
Early accounts of semantic bootstrapping (e.g. Berwick, 1985;
Wexler & Culicover, 1980) assumed perfect access to a single
meaning representation in the form of an Aspects-style Deep Structure already aligned to the words of the language. Yet, as we shallsee, semantic bootstrapping is sufficiently powerful that suchstrong assumptions are unnecessary.
Since, on the surface, languages differ in many ways—for example with respect to the order of heads and complements, and inwhether such aspects of meaning as tense, causality, evidentiality,
and information structure are explicitly marked—the meaning representations must be expressed in a universal prelinguistic conceptual representation, in whose terms all such distinctions are
expressable. The mapping must further be learned by general principles that apply to all languages. These general principles are often

117

referred to as ‘‘universal grammar”, although the term is somewhat
misleading in the present context since the model we develop is
agnostic as to whether these principles are unique to language orapply more generally in cognition.
A number of specific instantiations of the semantic bootstrapping theory have been proposed over the years. For example, ‘‘parameter setting” accounts of language acquisition assume,
following Chomsky (1981), that grammars for each natural language can be described by a finite number of finitely-valued
parameters, such as head-position, pro-drop, or polysynthesis(Hyams, 1986 and much subsequent work). Language acquisition
then takes a form that has been likened to a game of TwentyQuestions (Yang, 2006 Ch:7), whereby parameters can be set when
the child encounters ‘‘triggers”, or sentences that can only be analyzed under one setting of a parameter. For example, for Hyams(1986), the fact that English has lexical expletive subjects (e.g., it
in it rained) is unequivocal evidence that the pro-drop parameteris negative, while for others the position of the verb in simple
intransitive sentences in Welsh is evidence for head-initiality. Suchtriggers are usually discussed in purely syntactic terms. However,in both examples, the child needs to know which of the words isthe verb, which requires a prior stage of semantic bootstrappingat the level of the lexicon (Hyams, 1986:132–133).
Unfortunately, parameter setting seems to raise as many questions as it answers. First, there are a number of uncertainties concerning the way the learner initially identifies the syntactic
categories of the words, the specific inventory of parameters thatare needed, and the aspects of the data that ‘‘trigger” their setting(Fodor, 1998; Gibson & Wexler, 1994; Niyogi & Berwick, 1996).
Second, several combinatoric problems arise from simplistic searchstrategies in this parameter space (Fodor & Sakas, 2005). Here, wewill demonstrate that step-like learning curves used to argue forparameter-setting approaches (Thornton & Tesan, 2007) can be
explained by a statistical model without explicit linguistic
parameters.
A further variant of the semantic bootstrapping theory to be
discussed below postulates a second, later, stage of ‘‘syntactic
bootstrapping” (Braine, 1992; Gleitman, 1990; Landau &Gleitman, 1985; Trueswell & Gleitman, 2007), during which theexistence of early semantically bootstrapped syntax allows rapid
or even ‘‘one-shot” learning of lexical items, including ones forwhich the situation of utterance offers little or no direct evidence.
Early discussions of syntactic bootstrapping implied that it is alearning mechanism in its own right, distinct from semantic bootstrapping. However, we will demonstrate that these effects attributed to syntactic bootstrapping emerge naturally under the theory
presented here. That is, our learner exhibits syntactic bootstrapping effects (using syntax to accelerate word learning) without
the need for a distinct mechanism: the mechanism of semantic
bootstrapping is sufficient to engender the effects.
Although varieties of semantic bootstrapping carry considerable
currency, some researchers have pursued an alternative distributional approach (Redington, Chater, & Finch, 1998), which assumes
that grammatical structure can be inferred from statistical properties of strings alone. Many proponents of this approach invoke Artificial Neural Network (ANN) computational models as an
explanation for how this could be done—see Elman et al. (1996)
for examples—while others in both cognitive science and computerscience have proposed methods using structured probabilistic
models (Cohn, Blunsom, & Goldwater, 2010; Klein & Manning,
2004; Perfors, Tenenbaum, & Regier, 2011). The distributional
approach is appealing to some because it avoids the assumptionthat the child can access meanings expressed in a language of mind
that is homomorphic to spoken language in the sense defined
above, but inaccessible to adult introspection and whose detailed
character is otherwise unknown.

118

O. Abend et al. / Cognition 164 (2017) 116–143

There has been some success in using this kind of meaning-free
approach to learn non-syntactic structure such as word- andsyllable-level boundaries (Goldwater, Griffiths, & Johnson, 2009;
Johnson & Goldwater, 2009; Phillips & Pearl, 2014). However,attempts to infer syntactic structures such as dependency or constituency structure, and even syntactic categories, have been notably less successful despite considerable effort (e.g. Abend, Reichart,& Rappoport, 2010; Christodoulopoulos, Goldwater, & Steedman,
2010; Cohn et al., 2010; Klein & Manning, 2004, 2005). Within
the context of state-of-the-art natural language processing (NLP)
applications, ANN models that have no explicit structured representations have yielded excellent language modeling performance(i.e., prediction of probable vs. improbable word sequences) (e.g.,
Mikolov, Karafiát, Burget, Cernockỳ, & Khudanpur, 2010;
Sundermeyer, Schlüter, & Ney, 2012). They have also been used
to learn distributed word representations that capture some
important semantic and syntactic information (Mikolov, Yih, &Zweig, 2013). Yet the reason these models work as well as they
do in NLP tasks arises from the way they mix sentence-internal
syntax and semantics with pragmatics and frequency of collocation. Thus, they often learn to conflate antonyms as well as synonyms to similar representations (Turney & Pantel, 2010). This
representation creates problems for the compositional semantics
of logical operators (such as negation) of a kind that the child never
exhibits.

1.2. Overview of the learner
In this work, we take a close relation between syntax and compositional semantics as a given. We also follow the basic premise of
semantic bootstrapping that the learner is able to infer the meaning of at least some of the language she hears on the basis of nonlinguistic context. However, unlike some other versions ofsemantic bootstrapping, we assume that the available meanings
are at the level of utterances rather than individual words, and thatword meanings (i.e., the mapping from words to parts of the utterance meaning) are learned from such data.
We represent the meaning of an utterance as a sentential logicalform. Thus, the aim of the learner is to generalize from the input
pairs of an observed sentence and a possible meaning in order tointerpret new sentences whose meaning is unavailable contextually, and to generate new sentences that express an intended
meaning. Because of the limitations of available corpus annotations, the logical forms we consider are restricted to predicateargument relations, and lack the interpersonal content whose
importance in language acquisition is generally recognized. We
examine the nature of such content in Section 4.4, where we argue
that our model can be expected to generalize to more realistic
meaning representations.
Recent work suggests that the infant’s physically limited viewof the world, combined with social, gestural, prosodic, and othercues, may lead to considerably less ambiguity in inferring utterance meanings than has previously been supposed (Yu & Smith,
2012, 2013; Yurovsky, Smith, & Yu, 2013). Nevertheless, most contexts of utterance are likely to support more than one possible
meaning, so it is likely that the child will have to cope with a number of distracting spurious meaning candidates. We model propositional ambiguity in the input available to the learner byassuming each input utterance s is paired with several contextuallyplausible logical forms fm1 ; . . . ; mk g, of which only one is correct
and the rest serve as distractors. While these meaning representations are assumed to reflect an internal language of mind, ourmodel is general enough to allow the inclusion of various typesof content in the logical forms, including social, informationstructural, and perceptual.

Within this general framework, we develop a statistical learnerthat jointly models both (a) the mapping between components ofthe given sentential meaning and words (or phrases) of the language, and (b) the projection of lexical elements onto constituentsand sentences by syntactic rules. In earlier work, we defined thelearner and gave preliminary simulation results (Kwiatkowski,
Goldwater, Zettlemoyer, & Steedman, 2012); here we expand considerably on the description of the learner, the range of simulations, and the discussion in relation to human language
acquisition.
There has been considerable previous work by others on bothword learning and syntactic acquisition, but they have until very
recently been treated separately. Thus, models of crosssituational word learning have generally focused on learning either
word-meaning mappings (mainly object referents) in the absenceof syntax (Alishahi, Fazly, & Stevenson, 2008; Frank, Goodman, &Tenenbaum, 2009; McMurray, Horst, & Samuelson, 2012;
Plunkett, Sinha, Møller, & Strandsby, 1992; Regier, 2005; Siskind,
1996; Yu & Ballard, 2007), or learning verb-argument structures
assuming nouns and/or syntax are known (Alishahi & Stevenson,
2008, 2010; Barak, Fazly, & Stevenson, 2013; Beekhuizen, Bod,
Fazly, Stevenson, & Verhagen, 2014; Chang, 2008; Morris,
Cottrell, & Elman, 2000; Niyogi, 2002).4 Conversely, most models
of syntactic acquisition have considered learning from meaningfree word sequences alone (see discussion above), or have treatedword-meaning mapping and syntactic learning as distinct stages oflearning, with word meanings learned first followed by syntax(Buttery, 2006; Dominey & Boucher, 2005; Villavicencio, 2002).
Several previous researchers have demonstrated that correct(adult-like) knowledge of syntax (e.g., known part-of-speech categories or syntactic parses) can help with word-learning (Mellish,
1989; Göksun, Küntay, & Naigles, 2008; Ural, Yuret, Ketrez,
Koçbasß, & Küntay, 2009; Fazly, Alishahi, & Stevenson, 2010;
Thomforde & Steedman, 2011; Yu & Siskind, 2013), and a few(Alishahi & Chrupała, 2012; Yu, 2006) have gone further in showing that learned (and therefore imperfect) knowledge of POS categories can help with word learning. However, these models are nottruly joint learners, since the learned semantics does not feed backinto further refinement of POS categories.
By treating word learning and syntactic acquisition jointly, ourproposal provides a working model of how these two aspects oflanguage can be learned simultaneously in a mutually reinforcing
way. And unlike models such as those of Maurits et al. (2009)
and Beekhuizen (2015), our model learns from real corpus data,
meaning it needs to handle variable-length sentences and predicates with differing numbers of arguments, as well as phenomena
such as multiple predicates per sentence (including hierarchicalrelationships, e.g., want to go) and logical operators, such as negation and conjunction. To tackle this challenging scenario, we adopt
techniques originally developed for the task of ‘‘semantic parsing”(more properly, semantic parser induction) in computational linguistics (Kwiatkowski, Zettlemoyer, Goldwater, & Steedman,
2010; Kwiatkowski et al., 2011; Thompson & Mooney, 2003;
Zettlemoyer & Collins, 2005, 2007).
Our model rests on two key features which we believe to be
critical to early syntactic and semantic acquisition in children.
The first, shared by most of the models above, is statisticallearning. Our model uses a probabilistic grammar and lexicon
whose model parameters are updated using an incremental
learning algorithm. (By parameters here, we mean the probabil4
Most of these verb-learning models actually model argument structure generalization (e.g., predicting that if a verb has been seen in a double object construction, it
might also be acceptable in a dative construction). We do not address this type ofgeneralization directly in the model presented here. However, Kwiatkowski,
Zettlemoyer, Goldwater, and Steedman (2011) have developed lexical generalization
methods for similar models that could begin to address this problem.

O. Abend et al. / Cognition 164 (2017) 116–143

ities in the model; our model does not include linguistic parameters in the sense noted earlier of Chomsky (1981) and Hyams(1986).) This statistical framework allows the model to takeadvantage of incomplete information while being robust to
noise. Although some statistical learners have been criticizedfor showing learning curves that are too gradual—unlike thesudden jumps in performance sometimes seen in children(Thornton & Tesan, 2007)—we show that our model does not
suffer from this problem.
The second key feature of our model is its use of syntacticallyguided semantic compositionality. This concept lies at the heartof most linguistic theories, yet has rarely featured in previous computational models of acquisition. As noted above, many models
have focused either on syntax or semantics alone, with anotherlarge group considering the syntax-semantics interface only as it
applies to verb learning. Of those models that have considered bothsyntax and semantics for full sentences, many have assumed thatthe meaning of a sentence is simply the set of meanings of thewords in that sentence (Alishahi & Chrupała, 2012; Allen &Seidenberg, 1999; Fazly et al., 2010; Yu, 2006). Connor, Fisher,and Roth (2012) addressed the acquisition of shallow semantic
structures (predicate-argument structures and their semanticroles) from sentential meaning representations consisting of theset of semantic roles evoked by the sentence. Villavicencio(2002) and Buttery (2006) make similar assumptions to our own
about semantic representation and composition, but also assume
a separate stage of word learning prior to syntactic learning, withno flow of information from syntax back to word learning as inour joint model. Thus, their models are unable to capture syntactic
bootstrapping effects. Chrupała et al. (2015) have a joint learningmodel, but no explicit syntactic or semantic structure. The model
most similar to our own in this respect is that of Jones (2015),
but as noted above, the simulations in that work are more limited
than those we include here.
Much of the power of our model comes from this assumptionthat syntactic and semantic composition are closely coupled. To
implement this assumption, we have based our model on Combinatory Categorial Grammar (CCG, Steedman, 1996b, 2000, 2012).
CCG has been extensively applied to parsing and interpretationof unrestricted text (Auli & Lopez, 2011; Clark & Curran, 2004;
Hockenmaier, 2003; Lewis & Steedman, 2014), and has received
considerable attention recently in the computational literature
on semantic parser induction (e.g., Artzi, Das, & Petrov, 2014;
Krishnamurthy & Mitchell, 2014; Kwiatkowski et al., 2010, 2011;
Matuszek, Fitzgerald, Zettlemoyer, Bo, & Fox, 2012; Zettlemoyer& Collins, 2005, 2007).
This attention stems from two essential properties of CCG.
First, unlike Lexicalized Tree-Adjoining Grammar (Joshi &Schabes, 1997), Generalized Phrase Structure Grammar (Gazdar,
Klein, Pullum, & Sag, 1985), and Head-driven PSG (Pollard &Sag, 1994)—but like LFG (Bresnan, 1982) and the Minimalist program (Chomsky, 1995)—a single nondisjunctive lexical entry
governs both in situ and extracted arguments of the verb. Second,
the latter kind of dependencies are established without the overhead for the learner of empty categories and functional uncertainty or movement, of the kind used in LFG and the
Minimalist Program.
These properties of CCG, together with its low near-context-free
expressive power and simplicity of expression, have made it attractive both for semantic parsing, and for our purposes here.
Nevertheless, the presented framework can in principle be implemented with any compositional grammar formalism as long as(1) it allows for the effective enumeration of all possible syntactic/semantic derivations given a sentence paired with its meaning
representation; and (2) it is associated with a probabilistic modelthat decomposes over these derivations.

119

Using our incremental learning algorithm, the learner is trainedon utterances from the Eve corpus (Brown, 1973; Brown & Bellugi,1964) in the CHILDES database (MacWhinney, 2000), with meaning representations produced automatically from an existing
dependency annotation of the corpus (Sagae, Davis, Lavie,
MacWhinney, & Wintner, 2010). We use these automatically produced meaning representations as a proxy for the child’s actual
meaning representation in the hidden conceptual language ofmind. (Crucially, our model entirely ignores the alignment in thesedata between logical constants and English words, as if the sentences were in a completely unknown language.)
We evaluate our model in several ways. First, we test the learner’s ability to correctly produce the meaning representations ofsentences in the final Eve session (not included in the trainingdata). This is a very harsh evaluation, since the learner is trainedon only a small sample of the data the actual child Eve was exposed
to by the relevant date. Nevertheless, we show a consistentincrease in performance throughout learning and robustness tothe presence of distractor meaning representations during training.
Next, we perform simulations showing that a number of disparate
phenomena from the language acquisition literature fall out naturally from our approach. These phenomena include sudden jumps
in learning without explicit parameter setting; acceleration ofword-learning (the ‘‘vocabulary spurt”); an initial bias favoring
the learning of nouns over verbs; and one-shot learning of words
and their meanings, including simulations of several experiments
previously used to illustrate syntactic bootstrapping effects. The
success of the model in replicating these findings argues that separate accounts of semantic and syntactic bootstrapping are unnecessary; rather, a joint learner employing statistical learning overstructured representations provides a single unified account of
both.

2. Semantic bootstrapping for grammar acquisition
The premise of this work is that the child at the onset of
language acquisition enjoys direct access to some form of
pre-linguistic conceptual representations. We are not committed
to these representations taking any particular symbolic or
non-symbolic form, but for the compositional learning process toproceed, there must be some kind of structure in which complex
concepts can be decomposed into more primitive concepts, andhere we will abstractly represent this conceptual compositionality
using a logical language. Such universal semantics or conceptual
structure, broadly construed, has often been argued to be the mostplausible source for the universal learning biases that enablelanguage acquisition (Chomsky, 1965:27–30; Crain & Nakayama,
1987; Chomsky, 1995:54–55; Pinker, 1979; Croft, 2001;
Ambridge, Pine, & Lieven, 2014). The child’s task is therefore toconsider all the different ways that natural language allows chunksof meaning representation to be associated with lexical categories
for the language they are presented with.
Concretely, we address the following learning problem. Given a
corpus of transcribed utterances, paired with representations of
their meaning (henceforth, their logical forms), the goal of learning
is to induce a mapping between utterances (or transcribed text) tomeaning representations, so as to support the correct interpretation of unseen text, namely by assigning it correct logical forms.
The model represents two types of information: a (probabilistic) pairing between lexical items and their meaning representation, and a distribution over the syntactic derivations the learnerhas been exposed to (a probabilistic grammar). The sensitivity of
the infant learning language to statistical trends observed in thedata has been demonstrated in multiple levels of linguistic representation (Gómez & Maye, 2005; Mintz, 2003; Saffran, Aslin, &120

O. Abend et al. / Cognition 164 (2017) 116–143

Newport, 1996 inter alia) and is a key factor in this account ofsemantic bootstrapping.
Much of the difficulty in this setting stems from not being able
to directly observe the meaning of individual words and thederivation trees, as the only supervision provided is of sentential
meaning representations. In order to infer the probabilistic grammar and word meanings, the model entertains all possible derivations and word-meaning pairings, and weighs these based on theirlikelihood according to its hitherto acquired beliefs. This weighted
space of derivations is subsequently used to update the beliefs of
the learner when exposed to the next utterance.
2.1. Meaning representations
Meanings of sentences in CCG (and in our model) are expressedin first-order predicate logic. The example(1) a. ‘‘you like the doggies!”
b. like(you, doggies)
expresses a relation like between the entity you and the entity
doggies.
Of course, we do not believe that the child’s conceptual representation is as simple as this. As Tomasello (1999) has pointed
out, the content that the child is actually working with and that
actually provides it with the incentive to engage with language ishighly interpersonal and social—probably more like something
paraphrasable as ‘‘Mum is sharing my liking the doggies”. Nevertheless, our model needs to represent the child’s semantics somehow. For simplicity, we’ll use terms of a naive English-centric
lambda-calculus, and defer the question of what a more psychologically realistic human logical language might actually look likeuntil Section 4.4.
The lambda-calculus uses the k-operator to define functions.
These may be used to represent functional meanings of utterances
but they may also be used as a ‘‘glue language” to compose elements of first order logical expressions like the above. For example,
the function kxky.like(y, x) can be combined with the argument
doggies to give the phrasal meaning ky.like(y, doggies) (corresponding to the VP ‘‘like the doggies”) via the lambda-calculus operationof function application. In function application, the formal argumentx (introduced by the lambda notation) is replaced by doggies in theremainder of the expression.
Our model uses typed k-expressions, where each variable andconstant is given a semantic type. These types may either be
atomic—such as e for entities (e.g., you, doggies) and t for truth values—or they may be complex—for instance, sleep is a function fromentities to a truth value, of type (e, t), and like is a (Curried) function
from pairs of entities to a truth value, of type (e, (e, t)). Lambdacalculus operations such as function application and composition
are only allowed between forms of compatible types. For brevity,
we will often not state semantic types explicitly.
Despite the shortcomings of such an approach to semantic
representation, the logical forms we use do meet the minimal criteria of (1) not encoding word order information and (2) not
encoding information about the correct segmentation of theutterance’s semantic forms above the level of atomic semantic
symbols. For example, the logical form like(you, doggies) does
not determine the number of lexical items that will give rise tothis composite representation. The space of possible derivations
that can give rise to like(you, doggies) includes the possibilities
that there are four words in the sentence, one matching like,one matching you and one matching doggies, but also the possibility that the meaning is partitioned into other numbers ofwords, including the possibility that the whole meaning corresponds to a single word.

2.2. The grammar
Combinatory Categorial Grammar (CCG) is a strongly lexicalised
linguistic formalism that tightly couples syntactic types and corresponding semantic k-terms. Each CCG lexical item in the lexicon L
is a triplet, written

word ‘ syntactic category : semantic form:
Examples include5:

you ‘ NP : you
sleep ‘ SnNP : kx:sleepðxÞ
like ‘ ðSnNPÞ=NP : kxky:likeðy; xÞthe ‘ NP=N : kf :theðx; f ðxÞÞdoggies ‘ N : kx:dogsðxÞ
Syntactic categories may be atomic (e.g., S or NP) or complex(e.g., SnNP or ðSnNPÞ=NP). Slash operators in complex categories
define functions from the domain on the right of the slash to theresult on the left in much the same way as lambda operators doin the corresponding semantic forms. The difference is that thedirection of the slash operator defines the linear order of function
and argument in the sentence. For example, S=NP corresponds to a
constituent that along with an NP to its right forms a constituent
with the category S, while SnNP is similar but requires an NP to
its left to form an S. For example, in the English sentence ‘‘you
sleep”, the verb ‘‘sleep” has the latter category, SnNP. In a verbinitial language like Welsh, the corresponding intransitive verb
meaning ‘‘sleep” has the former category, S=NP.
CCG uses a small set of combinatory rules to concurrently combine constituents according to their syntactic types and compose
their semantic representations. Two simple combinatory rules
are forward (>) and backward (<) application:

X=Y : f

Y:g

) X : f ðgÞ ð>Þ

Y:g

XnY : f

) X : f ðgÞ ð<ÞGiven the lexicon above, the phrase ‘‘you like the doggies” canbe parsed using these rules as follows, where each step in the parseis labeled with the combinatory rule (> or <) as in Fig. 1a.
It is standard to write CCG parse trees as in Fig. 1a, with leaves
at the top and combinatory operations written next to constituent
demarcation lines. However, the same derivation could also
be written in phrase-structure form as in Fig. 1b (semantics
omitted).
A bit more needs to be said about transitive verbs. Unlike unary
intransitive predicates and the determiner categories, transitive
verbs in English, such as ‘‘like” in the example above, could in principle be associated with either of the two syntactic categories in(2), both of which support a derivation of the final logical form.
(2) a. like ‘ (SnNP)/NP: kxky.like(y, x)b. like ‘ (S/NP)nNP: kykx.like(y, x)
However, we will assume here that the universally permitted
set of transitive categories excludes cases of verb-medial categories (SVO or OVS) where, as in 2(b), the verb attaches first to
the semantic subject.6 The learner therefore considers only the following six possible categories/constituent orders for transitive verbs,
only one of which will work for English7:
5
Determiner semantics is represented as generalized quantifiers. Namely,
the(x,f(x)) should be read as the generalized quantifier the over x, applied to f(x).
6This assumption rules out verb-medial ergative languages but can be circumvented by subcategorizing the subject and object NPs based on their case.
7
We assume, following Baldridge (2002), that free word-order languages simply
have more than one of these categories (which may of course be schematized).

O. Abend et al. / Cognition 164 (2017) 116–143

(a)

(b)

Fig. 1. Parse trees for the sentence ‘‘you like the doggies” using the grammar in
Section 2.2. (a) using standard CCG notation. (b) using phrase-structure notation,with semantics omitted.

(3) a. SOV ‘ (SnNP)nNP: kykx.like(x, y)b. SVO ‘ (SnNP)/NP: kykx.like(x, y)c. VSO ‘ (S/NP)/NP: kxky.like(x, y)d. VOS ‘ (S/NP)/NP: kykx.like(x, y)e. OVS ‘ (S/NP)nNP: kykx.like(x, y)f. OSV ‘ (SnNP)nNP: kxky.like(x, y)Finally, we note that our model also includes CCG combinatory
rules of forward (> B) and backward (< B) composition:

X=Y : f

Y=Z : g

) X=Z : kx:f ðgðxÞÞ ð> BÞ

YnZ : gXnY : f) XnZ : kx:f ðgðxÞÞ ð< BÞ

These rules are crucial to the CCG analysis of unbounded
dependency-inducing constructions such as relativization, towhich we return in the discussion of later learning in Section 4.4.
2.3. The probabilistic model
The goal of the presented model is to learn a form-meaning
mapping that can support the correct interpretation of unseen
text. We assume that this mapping involves a latent syntactic
structure that mediates between phonological/orthographic andsemantic form. Given the uncertainty involved in grammar induction (as a finite number of examples does not uniquely determinea CCG grammar, or any other plausible class of grammars), we
formulate the problem of acquisition as one of probabilistic
inference.
There are two ways to construe the use of probabilistic
modeling. In one, probabilities serve to make rational inferences in an uncertain environment. In this case, the target
grammar can be formulated without the use of probabilities,
while the probabilistic model represents the ‘‘belief” of themodel in each of the possible non-probabilistic hypotheses.
Alternatively, the target grammar itself is assumed to be probabilistic, either because it captures information that is probabilistic in nature or because the model does not have access121

to some of the information that would be needed to disambiguate (for example, semantic distinctions not represented in
the logical forms we use). Our model and results are consistent with either interpretation; we will return to discuss thispoint in Section 4.
We take a Bayesian inference approach and explore learning
through an idealized rational learner that induces the grammar
most supported by the observed data given the assumptions madeby the model. We depart from this idealized setting only in assuming that learning is done incrementally, so that the learner has noaccess to previously observed examples beyond what has alreadybeen learned from them (i.e., the updated parameters of the statistical model). Using an incremental algorithm makes it easy to
examine how the learner’s grammar and behavior develop overtime as examples are presented, and is of course more psychologically plausible than a batch algorithm.
As discussed above, our model takes as input a sequence ofsentences, each paired with one or more possible meanings.
For the moment, assume each sentence s is paired with a single meaning m. The learner also maintains a probability distribution over model parameters H. These parameters specify theprobabilities of using or combining lexical items into phrases
according to different rules, and can be thought of as the learner’s grammar (including the lexicon, as CCG is a lexicalized
formalism). The distribution over H thus represents the learner’s belief about which grammars are most plausible given
the input so far.
Before observing a particular ðs; mÞ pair, the learner has someprior probability distribution over the grammar, PðHÞ. At the
beginning of the learning process, this prior will be entirely flat,
and will assign equal probability to symmetric options (e.g.,verb-final vs. verb-initial) as the learner doesn’t know anything
specific about the language being learned.8 However, the priorwill be updated with each new example. Upon observing an
ðs; mÞ pair, the learner infers the probabilities of all possible syntactic derivation trees t that could provide a derivation of the
ðs; mÞ pair, along with a posterior distribution over H. That is,
the learner infers a distribution Pðt; H j s; mÞ, which can be
expressed using Bayes’ rule as:Pðt; H j s; mÞ ¼Pðs; m; t j HÞPðHÞ
Pðs; mÞ

ð1Þ

As learning progresses, the prior for each new example is in fact
the posterior distribution learned after observing the previous
example, so over time the prior will become much more peaked,
assigning high probability to lexical items and rules used frequently in the previously inferred derivations.
For the model to be complete, we also need to specify how tocompute the likelihood Pðs; m; t j HÞ, which determines how probable different derivations are under a particular grammar. (For any
ðs; mÞ pair, Pðs; mÞ is a constant that simply normalizes the posterior distribution to sum to 1, so we don’t need to define it
explicitly.)
As is usual in Bayesian models, we will define the likelihoodusing a generative formulation, i.e., we will define a probabilistic
process that describes how ðs; m; tÞ triples (derivations) are
assumed to be generated given a particular set of parameters H.
This process will consist of multiple steps; by multiplying together
the probabilities at each step we can determine the probability of
the final complete derivation Pðs; m; t j HÞ. The learner that we present in Section 2.4 incrementally changes the parameters H to
8
There is nothing in our model to exclude asymmetric priors, such as those
proposed by Culbertson, Smolensky, and Wilson (2013). We assume a flat initial prioras it is the most conservative option.

122

O. Abend et al. / Cognition 164 (2017) 116–143

increase the probabilities of derivations that support the ðs; mÞpairs seen during training.9
Consider now a particular ðs; m; tÞ derivation, such as the one
given in Fig. 1 for ‘‘you like the doggies”: like(you, the(x, dogs(x))).
The probability of such a derivation will have three parts. First,
the probability of all of the function applications needed to decompose each parent node into each child combination of nodes. Theseare essentially the rules of the syntax. Second, the probability withwhich each of the resulting leaf syntactic category child nodes (e.g.,
NP or NP=N) is assigned a given meaning (e.g., you or kf.the(x, f(x))).
Third, the probability with which each meaning corresponds to aword in the sentence (e.g., ‘you’ or ‘the’).
Formally, let w1 ; . . . ; wn be the words in the sentences, m1 ; . . . ; mn be their meaning representations (i.e., the meaningsof each leaf in the derivation tree), and c1 ; . . . ; cn be their syntactic
categories. The derivation is generated as follows:
1. Generate each syntactic rule in the derivation, with probability
Pðc j aÞ for a rule a ! c with parent category a and child categories c. Each parent category X either generates a pair of childcategories through inverse application or composition (e.g., X
may generate X=Y and Y; X=Z may generate X=Y and Y=Z; See
Appendix B for the formal definition of this procedure), or generates a leaf by generating the symbol XLEX . We assume allderivations start with the START symbol, but for brevity omit
this and the XLEX categories from the example derivations
shown. The probability of this step is given as:

PSYNTAX ðtÞ ¼

YPðc j aÞ

ð2Þ

a!c2t

2. For each leaf syntactic category ci in c1 ; . . . ; cn , generate the corresponding meaning representation mi with probabilityPðmi j ci Þ. The probability of this step is given as:

PMEANINGS ðm j tÞ ¼

nYPðmi j ci Þ

ð3Þ

i¼1

3. For each mi , generate the corresponding word wi with probability Pðwi j mi Þ. The probability of this step is given as:

PWORDS ðs j mÞ ¼

nYPðwi j mi Þ

ð4Þ

i¼1

Putting these steps together, we get the overall probability of a
derivation:

Pðs; m; tÞ ¼ PSYNTAX ðtÞ  PMEANINGS ðm j tÞ  P WORDS ðs j mÞ

ð5Þ

For instance, the probability of the derivation of ‘‘you like thedoggies” given in Fig. 1 is the product of the following three terms:
P SYNTAX ðtÞ ¼ PðS j STARTÞ  PðNP; SnNPjSÞ  PðSnNP=NP; SnNPjSnNPÞPðNP=N; NjNPÞ  PðNPLEX j NPÞ
Pð½SnNP=NPLEX j SnNP=NPÞPð½NP=NLEX j NP=NÞ  PðNLEX j NÞ
P MEANINGS ðmjtÞ ¼ Pðyou j NPLEX Þ  Pðkxky:likeðy; xÞ j ½SnNP=NPLEX ÞPðkf :theðx; f ðxÞÞ j ½NP=NLEX Þ  Pðkx:dogsðxÞ j NLEX Þ
P WORDS ðsjmÞ ¼ Pðyou j youÞ  Pðlike j kxky:likeðy; xÞÞPðthe j kf :theðx; f ðxÞÞÞ  Pðdogs j kx:dogsðxÞÞ
9
Since the syntactic tree t for any particular ðs; mÞ pair is never directly observed,
the likelihood term favors values of H that assign high probability to the set of possible
derivations t ðs;mÞ of the ðs; mÞ pair in question—Appendix B describes how the space ofall possible derivations is defined. When multiple ðs; mÞ pairs are observed, thelikelihood favors H that assign high probability to the set of all derivations of allobserved pairs. This goal can be achieved by assigning high probability to derivations
with subparts that can be used to explain many different ðs; mÞ pairs. Assigning highprobability to an idiosyncratic parse could raise the probability of t ðs;mÞ for a particularðs; mÞ pair, but will steal probability mass from other parses that are more generally
applicable, thus making the possible derivations for all other ðs; mÞ pairs less likely.

This model is essentially a probabilistic context-free grammar(PCFG). While naive PCFGs impose conditional independence
assumptions that are too strong to characterize natural language,
they can be augmented to overcome this problem (Charniak,
1997; Collins, 1997). PCFGs are a simple and well-understood classof models with standard parsing and learning algorithms, and are
powerful enough to do well in prediction of both phrasestructure trees (Collins, 1997) and CCG derivation trees(Hockenmaier & Steedman, 2002) in naturalistic settings. Future
work will explore the use of richer families of distributions.
In order to complete the model, we still need to define how the
distributions used in Eqs. (2)–(4) are calculated: Pðc j aÞ; Pðm j cÞ,
and Pðw j mÞ. First, consider Pðc j aÞ, the distribution over possible
expansions c given a syntactic production head a. Notice that this
distribution needs to be defined over an infinite set of possible
expansions, since the learner does not know ahead of time which
expansions will be needed for their language. Recent probabilistic
models of language acquisition (Alishahi & Stevenson, 2008, 2010;
Feldman, Griffiths, Goldwater, & Morgan, 2013; Frank, Feldman, &Goldwater, 2014; Goldwater et al., 2009), have demonstrated that
Dirichlet Processes can be used to successfully model such infinite
distributions, so we adopt that approach here. When applied to our
problem (see Appendix A for details), the Dirichlet Process defines
the probability of a particular expansion given the production headasPðc j aÞ ¼

na!c þ asyn Ha
na þ asyn

ð6Þ

where na!c is the number of times the rule a ! c has been used previously in a derivation, na is the number of times any rule expandinga has been used, and asyn is a fixed parameter of the model which
determines the learner’s tendency to infer derivations containing(any) new rule that has not been used previously. Ha , another fixedparameter of the model, is a prior distribution over rules: of all therules headed by a that have not been used before, which ones are
more likely to be used?
Notice that under this definition, the more a rule has been used
relative to other expansions of the same parent, the higher its
probability will be. Over time, rules with higher probability will
be used more, incurring higher counts, which raises their probability even more: a virtuous cycle of learning. However, even rulesthat have never been used before (with na!c ¼ 0) have some
non-zero probability associated with them, as determined by thevalues of asyn and Ha . We define Ha without any particular bias
towards any of the CCG operators, or any of the atomic syntactic
categories. However, we do include a bias towards simpler syntactic categories (e.g., expanding an NP into ðNP=N; NÞ would be a priori more likely than expanding it into ðNP=ðN=NÞ; N=NÞ).
The remaining two distributions Pðm j cÞ (the distribution overleaf meaning representations given the syntactic category) andPðw j mÞ (the distribution over wordforms given a meaning representation m) are also defined using Dirichlet Processes. Again,
these provide well-defined distributions over the infinite sets ofpossible meaning representations and wordforms, assigning higher
probability to meanings and words that have been seen more frequently, but non-zero probability to novel meanings and words.
The prior distribution over unseen meanings prefers simpler logical expressions (those containing fewer logical constants), andthe prior distribution over unseen words prefers shorter words.
See Appendix A for details.
2.4. The learning algorithm
The previous subsections described the input, internal representations, and probabilistic model used by our learner. This

123

O. Abend et al. / Cognition 164 (2017) 116–143

subsection explains how the learner processes the input data inorder to determine the possible derivations of each (sentence,
meaning) pair and update its knowledge of the parameters H on
the basis of the input and its derivations.
At a high level, the learning algorithm is straightforward. The
corpus contains sentences s1 . . . sN , and we assume for the momentthat each si is paired with a single unambiguous meaning representation mi . The learner processes the corpus incrementally, at eachstep considering the pair ðsi ; mi Þ and computing the set of possible
derivations ti for this pair. Roughly speaking, a single derivation
can be found by recursively splitting si and mi into smaller andsmaller chunks that, when recombined using the CCG applicationand composition rules, will yield the original input pair. By considering all possible splits at each step in this process, all possible
derivations can be found and their probabilities computed basedon the learner’s current beliefs about H and the probabilistic model
described in Section 2.3.10 Details of the splitting process, including
how the possible syntactic categories are determined for each chunk,can be found in Appendix B.
Next, the learner updates its beliefs about H. Each derivation inti will include binary (combinatory) rules as well as unary rulesthat generate the leaf syntactic categories, meaning representations, and wordforms. The learner updates its probabilities forany rules used in ti , weighting the number of times a rule is usedby the probability of the derivation(s) in which it is used. In otherwords, the learner will increase the probability of a rule more if it isused frequently, or in a derivation that has high probability according to the current model parameters. It is also important to accountfor the fact that early on, the learner’s prior is based on little evidence, whereas later it is based on much more evidence. We therefore assume that early in learning the learner makes larger
updates, weighting the evidence from the current example moreheavily relative to the model’s previously learned prior. The rate
at which learning drops off is controlled by an additional fixedparameter, the learning rate. A more gradual change in the learning
rate means the learner will continue to make large updates forlonger, effectively forgetting more about early examples and taking
longer to converge. Details of the update procedure can be found in
Appendix C.11
The only change needed in order to handle the more realistic
condition where the child/learner has to consider more than one
possible meaning for each sentence is that the set of derivations
considered is the union of all possible derivations for the current
sentence paired with each possible meaning. In this scenario, rulesthat are used in derivations for the incorrect meaning may gain
some probability mass, especially early on when the learner has little evidence of the correct rules. However, they will quickly beswamped by the correct rules, for the same reason that rules used
in incorrect derivations of the correct meaning are. That is, the correct rules apply consistently (and therefore frequently) across awide range of sentences, leading to increasingly high probabilities
for those rules and a virtuous cycle of learning. As we shall see,even when there is an unknown word in the sentence, the learner’s
increasing knowledge about the rules that are typically used inother sentences can lead to very rapid learning of new words (boththeir meanings and syntax) later on in learning.
To illustrate the learning algorithm, consider a very simple
example in which the child’s first input is ‘‘more doggies”, which
we will assume, simplifying as usual, that she understands as
more(dogs). Upon hearing this sentence, the child inversely applies
the CCG combinators to retrieve every derivation possible under10
We efficiently store and compute probabilities over the set of possible derivations
by using a ‘‘packed forest” representation of the kind used in chart parsing incomputational linguistics.
11
We show later that within broad limits, the learning rate has no interesting
effects on the overall performance of the learner.

CCG that yields the input pair ‘‘more doggies”: more(dogs). In thiscase, there are three possible derivations:

The following set of candidate lexical entries can be read off thethree derivations in (4):(5) a. more ‘ NP/N: moreb. more ‘ N: dogs
c. doggies ‘ NP/N: mored. doggies ‘ N: dogs
e. more doggies ‘ NP: more(dogs)
Since this is the first utterance the child hears, the correct lexical entries (a, d) and incorrect entries (b, c) are completely symmetrical and therefore equiprobable: if the child wants to say
more she is as likely to choose ‘‘dogs” as ‘‘more”. Lexical entry (e)will have lower probability because the priors over wordforms
and logical forms favor shorter and simpler entries.
Now suppose the child encounters a second utterance, ‘‘morecookies”, which will have three derivations and five possible lexical
entries analogous to those for ‘‘more doggies”, with the correct
derivation being:

In this case the correct derivation will receive higher probability
than the incorrect ones, due to the child’s previous experience with‘‘more doggies”. In particular, because ‘‘more”: more has occurredalready in a possible derivation, Pðmore j moreÞ will be higher thanPðmore j cookiesÞ or Pðcookies j moreÞ, and this will give a higher
probability overall to the correct derivation, even though at thispoint there is no preference based on syntax. That is, prior toobserving ‘‘more cookies”, noun-initial and noun-final NPs haveboth been seen once in equiprobable derivations, so
PðN; NP=N j NPÞ is equal to PðNPnN; N j NPÞ. Importantly for futurelearning, however, the child is now able to update her syntactic
probabilities based on this example. Because the derivation in (6)has higher probability than the competing derivations, theNP ! NP=N N rule will be updated to receive higher probability
than the alternative NP ! N NPnN, and in future examples thiswill allow the child to use syntax to help infer the meanings of

124

O. Abend et al. / Cognition 164 (2017) 116–143

novel words. Our simulations in Sections 3.4 and 3.5 suggest that
this effect leads to an acceleration of learning over time, and Section 3.7 shows that the same effect can explain results from classic
syntactic bootstrapping experiments, where children use word
order to correctly learn the meaning of a novel verb used with
two familiar nouns (e.g., the man daxed the baby).
It should be noted that, due to the incremental nature of ourlearning algorithm, observing an utterance like ‘‘more cookies”
does not cause the model probabilities for dogs to be updated;i.e., the probability that dogs is realized as ‘‘doggies” will still beequal to the probability that it is realized as ‘‘more”. This behavior
would differ with a batch learning algorithm, since ‘‘more cookies”
provides evidence that more is realized as ‘‘more”, so that in ‘‘more
doggies”, dogs is more likely providing the semantics for ‘‘doggies”.
In our incremental algorithm, the parameters for any particularword are only updated when that word is seen. In that sense, themodel built by the incremental learner is an approximation towhat would be learned by an exact batch learner.

Table 1
Navigation table for the presented simulations and main effects.
SectionFigure(s)Simulation and main effect

3.2

1

Parsing accuracy improves over time; performance isrobust to propositional uncertainty

3.3

2–5

The model learns word ordering: that transitive
sentences are SVO, and that determiners andprepositions are pre-nominal. Steep learning curves are
exhibited despite having no explicit parameter-setting

3.4

6–9

The model demonstrates one-shot learning of verbs,
nouns, determiners and prepositions in later stages ofacquisition. These results can be seen as a naturalistic(corpus-based) analogue to the carefully controlled
simulations reported in Figs. 15–17, and therefore as aseparate illustration of how learned syntactic
knowledge can help accelerate word learning (the‘‘syntactic bootstrapping” effect; Gleitman, 1990)

3.5

10, 11

The model exhibits acceleration in the learning pace oftransitive verbs and of nouns, similar to the‘‘vocabulary spurt” observed in children (Reznick &Goldfield, 1992). Again, these results can be viewed as(partly) due to syntactic bootstrapping

3.6

12–14

The model obtains a larger production vocabulary ofnouns than of verbs, despite not being explicitly biased
to do so (Gentner, 1982; Tomasello, 1992:210). Moreover, in the presence of distractors, verbs are initially
under-represented in the learner’s production vocabulary (compared to nouns), relative to what can bepredicted from the verb to noun input ratio in thechild-directed utterances (Gentner & Boroditsky, 2001
and references therein)

3.7

15–17

The model simulates results from classic syntactic
bootstrapping experiments. It performs one-shotlearning, using its acquired knowledge of syntax todisambiguate between possible meanings for atransitive verb (following Fisher et al., 1994) andbetween noun-like and preposition-like meanings for anovel word (following Fisher et al., 2006)

2.5. Generating meanings for novel sentences
To evaluate our account of language learning, we apply the
resulting parser to unseen sentences and ask it to supply the corresponding meaning representation. Because of the generative formulation of the model, there is a straightforward probabilistic
interpretation of this task. In particular, given an input sentence s
and the model’s current estimate of the parameters H, we predictthe most probable, i.e., maximum a posteriori (MAP), meaning representation m and syntactic tree t  :t  ; m ¼ argmax Pðm; t j s; HÞð7ÞPðm; t; s j HÞPðs j HÞð8Þ

¼ argmax Pðm; t; s j HÞ

ð9Þ

m;t

¼ argmaxm;tm;twhere Eq. (9) follows from (8) because Pðs j HÞ doesn’t depend on mor t.
As the model is an instance of a PCFG, we use the standard probabilistic CKY algorithm for computing the most likely ðm; t; sÞ triplet given the parameters of the model.12 We note that the modelis unable to predict m if s contains unknown words, as the logicalconstants used to represent them can be any of an infinite set of
unknown items. Instead, the model predicts the most likely shell logical form, where anonymous but semantically typed place-holders
replace unseen constants. For example, suppose the model sees thesentence ‘‘Mary blicks John” with no paired logical form (corresponding to a situation where the child hears the sentence but cannot infer the semantics from non-linguistic context). If it has learned
enough about the syntax and semantics of other verbs and about themeaning of ‘‘Mary” and ‘‘John”, then it will be able to infer the meaning of the sentence as PLACE HOLDERðMary; JohnÞ, indicating that it
knows ‘‘blick” is a verb taking two arguments in a particular order,
but does not know the specific lexical meaning of ‘‘blick”. We also
investigate learning of this kind, using artificial unseen words.
3. Simulations
We conduct a range of simulations with our model, looking atfour main types of effects: (1) learning curves, both in terms ofthe model’s overall generalization ability, and its learning of specific grammatical phenomena; (2) syntactic bootstrapping effects,
12
Strictly speaking, the model is an infinite PCFG, which (unlike a standard PCFG),
places some probability mass on unseen expansions. However, such expansions are
excluded when predicting the MAP representation.

where previously acquired constructions accelerate the pace atwhich words are learned (we show overall trends and simulate
specific behavioral experiments); (3) one-shot learning effects,
showing that the model is able to infer the meaning of newly
observed words, given that enough about the syntax and the restof the words in the sentence is known; (4) findings as to the relative pace of learning of nouns and verbs.
Our simulations are summarized in Table 1 and described indetail below, following the description of the input data.
3.1. Input to the learner
3.1.1. Corpus
We know of no corpus of child-directed utterances manuallyannotated with compatible logical forms. Instead, we use the Eve
corpus (Brown & Bellugi, 1964) in the CHILDES database(MacWhinney, 2000) and construct the logical representations
semi-automatically based on the dependency and morphosyntactic category annotations of Sagae et al. (2010), using the general
method of Çakıcı (2005) and Ambati, Deoskar, and Steedman(2017).
The Eve corpus contains 20 roughly two-hour sessions, two permonth, collected longitudinally from a single child aged 18–
27 months. The dependency annotations were converted to logicalforms using a manually defined mapping that addresses major
constructions, such as verb-argument structure, coordination,
WH and yes/no questions, copulas and auxiliaries. The mapping
covers 41% of the utterances in the corpus, yielding 5831 utterances. About a quarter of the discarded utterances consist of oneword interjections such as ‘‘hmm” or ‘‘yeah”. We further filteredO. Abend et al. / Cognition 164 (2017) 116–143

125

Fig. 2. The model is trained by incrementally viewing sentences from the first i (x-axis) sessions, each paired with a correct meaning and up to six distractor meanings; it isthen tested on its accuracy (y-axis) at correctly assigning full sentence-level meanings to unseen test set sentences, using the 20th session as a test set. In (a), sentences
containing any previously unseen word are automatically marked as incorrect (since no specific semantics can be assigned to that word); in (b), the model can guessplaceholder meanings for unseen words, and receives credit if the rest of the semantic form is correct.

out sentences of 10 words or more, and of 10 or more extractablesub-expressions, leaving a final corpus of 5123 utterances. We
divide this into a training set of 4915 utterances (sessions 1–19),
and a held-out test set of 208 utterances (session 20). Some example sentence-meaning pairs can be found in Appendix D.13
This is the only dataset of its kind that we know of. However, it
should be noted that it constitutes what we estimate to be lessthan 2% of the data that the actual child Eve was learning from overthis same period in order to achieve the competence implicit insession 20, the held-out test set. It will become apparent belowthat this dataset is only just large enough to make stable and convergent learning possible, given our current model and learningalgorithm.

3.1.2. Propositional uncertainty
In order to relax the overly optimistic assumption that the childis exposed to entirely unambiguous situations, we add someuncertainty as to the correct logical form. We do so by pairing eachutterance with several logical forms, only one of the which is thecorrect one. Rather than choosing distractor logical forms at random, we follow Fazly et al. (2010) in choosing distractors fromnearby utterances in the corpus, so they are likely to share somesemantic content with correct logical forms (as they were utteredin a similar situation). In particular, distractors are taken to be thelogical forms of the w utterances preceding and following the target training example, where w 2 f0; 1; 2; 3g. We hence explore four
settings: one with no propositional uncertainty (target logical form
alone), and three with increasing amount of propositional uncertainty (target plus two, four and six distractors).

3.2. Parsing unseen sentences
We begin by evaluating our parser’s learning performance bytraining it on varying amounts of data, from 1 to 19 sessions. After
training on the first i sessions (for i ¼ 1 . . . 19), we test the learner’sability to parse the sentences in the held-out test file (session 20).
We use a single test file rather than testing each time on sessioni þ 1 to isolate the changes in performance due to learning, rather
than confounding results with possible changes in the test dataover time.
13
The derived corpus is available online through the corresponding author’s
website.

For each utterance in the test file, we use the parsing model topredict the MAP estimate for m and compare this to the target
meaning. As described in Section 2.5, when a word has never been
seen at training time our parser has the ability to ‘‘guess” a shell
logical form with placeholders for unseen constant and predicate
names. Fig. 2 presents the prediction accuracy, i.e., the proportionof utterances for which the prediction m matches the target
meaning with and without word-meaning guessing.
The low performance at all levels of uncertainty is largely due tothe sparsity of the data with 21% of all training sentences containing a previously unseen word. Nevertheless, while increasing the
propositional uncertainty slows the learning down, the learnershows increasing performance over time in all cases. This is anindication of the model’s robustness to noise, a major issue of concern in computational models of acquisition (Siskind, 1996; Yang,
2002).
3.3. Learning word order
We next examine the model’s learning of syntax, starting withtransitive verbs. There are eight possible ‘‘word orders” (categories
including semantic forms) that a transitive may be associated with,
two of which are only applicable to ergative languages, and are
ruled out by stipulation (see Section 2.2). The proposed model does
not have a parameter that directly represents its belief as to whichof the categories is the correct one. However, the model does maintain a distribution over syntactic expansions and logical forms,
which we use to compute the model’s learned prior distribution
over the six remaining transitive verb categories (where the model’s beliefs after seeing the training data are its prior over the categories used to parse any new utterance).
Fig. 3 presents the relative probability for the transitive verb
categories in the various settings. In all cases, the correct SVO category is learned to be the most probable upon convergence. In thethree cases with propositional uncertainty, we see that the SVO
category receives very little probability after 100 iterations (after
an initial probability of 16). The model is, however, able to recover,and eventually the correct SVO order is learned to be the mostprobable. However, in the 2 distractor setting the SVO category isnever overwhelmingly favored over the other categories (as it isin all the other settings). This stems from the difficulty in setting
a learning rate that works well across many different experimental
settings given the small size of the training set. Consequently, inthis case the algorithm does not manage to recover from its initial

126

O. Abend et al. / Cognition 164 (2017) 116–143

Fig. 3. Learning that English is an SVO language. Plots show the relative posterior probability assigned by the model to the six possible categories of transitive verbs. The xaxis is the number of training utterances seen so far. In the 2 distractor setting, the failure to assign most of the probability mass to the correct SVO category is an artefact of
the small dataset (see text).

under-estimation of the relative probability of SVO, since thelearning rate is too steep to allow effectively weighing in further
evidence observed later in learning. To confirm this, we experimented with a more gradual learning rate (details in AppendixC), and present the relative probabilities of the six categories forthis setting in Fig. 4, which indeed shows a clear preference for
SVO.

Results with a few other values for the learning rate present
similar trends with some changes in the details. Under all settings
the correct grammatical categories are favored (and with determiners and prepositions, overwhelmingly so) and the learning ofindividual words follows similar patterns to those described in
the following section. Changes in details include more fluctuating
learning curves when the learning rate is more moderate, andsomewhat different parameter values upon convergence.
To show that the model also learns the correct word order forother categories, we examined determiners (Fig. 5) and prepositions (Fig. 6), both in the high-uncertainty settings.14 (It is uninformative to look at nouns since the logical form in this case uniquelydetermines the syntactic category, namely N.) Results show that themodel quickly converges to the correct options (pre-nominal rather
than post-nominal determiners, and prepositions rather than postpositions), even in the presence of high propositional uncertainty.
In fact, the learning curves are even steeper than for verbs, despite
having no explicit ‘‘head-direction” parameter to set.

3.4. Learning curves of individual words
We next examine the learning curves for individual words,
again starting with transitive verbs. To do so, we estimate theprobability Pðv; SVO j mv Þ (henceforth, ‘‘correct production
Fig. 4. The relative probability of the six categories considered for transitive verbs
in the 2 distractor setting, as a function of the number of seen utterances, where thelearning rate is modified to be more gradual. The correct SVO category is learned
quickly and effectively.

14
We consider only prepositions that introduce an argument rather than an
adjunct. Adjunct prepositions have different syntactic and semantic types and arehence excluded by conditioning on the logical form.

O. Abend et al. / Cognition 164 (2017) 116–143(a) Syntactic disposition for determiners, 4 distractors

127(b) Syntactic disposition for determiners, 6 distractors

Fig. 5. Learning word order for determiners. Plots show the relative probability of generating a pre-nominal vs. a post-nominal determiner in the (a) 4 distractor and (b) 6
distractor setting, as a function of the number of training utterances seen.

(a) Syntactic disposition for prepositions, 4 distractors(b) Syntactic disposition for prepositions, 6 distractors

Fig. 6. Learning word order for prepositions. Plots show the relative probability of generating a preposition vs. a postposition in the (a) 4 distractor and (b) 6 distractor setting,as a function of the number of training utterances seen.

probability” or CPP) for a given transitive verb form v and its associated logical form mv . This probability is a proxy to the production
beliefs of the child, i.e., her ability to utter the correct word andsyntactic category, given an intended meaning. This is a strict measure, as it requires learning both the correct logical form and thecorrect syntactic category for the verb. We condition on the logicalform instead of the word form in order to avoid penalizing homographs, which are particularly common in our model, which e.g.
treats the word ‘‘run” in ‘‘they run”, ‘‘run along now”, and ‘‘I wantto run” as having three unrelated syntactic and logical forms whichjust happen to have the same surface form. This means that even a
perfect learner would have Pðmthird person plural run j runÞ < 1, simply
because some probability mass must also be reserved forPðmimperative run j runÞ, and this effect would vary in unpredictableways for different word forms, making it difficult to assess the success of learning. By contrast, in our training data each logical form
uniquely determines the correct syntactic type and word form, sothat a perfectly successful learner should always approach a CPP
of 1.
Fig. 7 shows how the CPP varies over the course of training for a
few transitive verbs, both frequent (‘‘move”, ‘‘want”, ‘‘fix”) and lessfrequent (‘‘cracked”, ‘‘needs”). With no propositional uncertainty,
all of these verbs are learned rapidly within a short timeframe after
they are encountered. In the cases of 4 and 6 distractors, learning ofword order is more gradual, revealing how inaccurate beliefs about

the structure of the language influence the learning of specificverbs. Indeed, verbs that are first observed in an earlier stage oflearning (e.g., ‘‘want”, ‘‘fix”, ‘‘move”) no longer exhibit one-shotlearning: instead, they are learned only gradually (and even seemto be ‘‘un-learned” on occasion; though the model is able torecover from the error).
Interestingly, the model’s knowledge about these words canactually improve even without seeing them again (see utterances 500–1000 in the 4 and 6 distractor cases) as its implicit
higher-order beliefs about grammatical structure (e.g., SVO
word order) change. As the model learns more about the language, its acquired knowledge also provides a strong priorguiding the interpretation of utterances containing new verbs.
This leads to much more rapid learning of verbs first observedlate in learning (‘‘cracked”, ‘‘needs”). There are two ways inwhich prior learning can help: first, knowing the meanings of(some) other words in the sentence reduces the uncertainty
about the meaning of the verb. Second, as we demonstrate
in Section 3.7, even if no other words in the sentence are
known, simply having high prior probability associated withrules instantiated by SVO categories will help to disambiguate
the verb’s meaning: syntactic bootstrapping at work. In thenaturalistic setting of this simulation, these two sources ofknowledge are inextricably linked; Section 3.7 isolates the syntactic effect by simulating a controlled laboratory experiment.

128

O. Abend et al. / Cognition 164 (2017) 116–143

Fig. 7. Learning curves showing the correct production probability for verbs of different frequencies in the (top) 0-distractor, (middle) 4-distractor, and (bottom) 6-distractor
settings, as a function of the number of utterances observed. (Results from the 2 distractor setting in this case are uninformative since the model fails to learn SVO word orderwell: see Fig. 3(b) and associated discussion.) Plots on the right zoom in on the first stage of learning, and also show horizontal ‘‘timelines” for each word with tickmarks
indicating, for each 20-utterance block, whether the word occurred in that block. (The color and top-to-bottom order of the timelines matches the words in the legend.) Largejumps in the plots generally correspond to observations in which the given verb appears, while smaller changes can occur due to fine-tuning of higher-order model
parameters (e.g., the overall probability of transitive verbs having SVO syntax). The frequencies for these verbs are: f ðmoveÞ ¼ 18; f ðwantÞ ¼ 70; f ðcrackedÞ ¼ 1;
f ðneedsÞ ¼ 3; f ðfixÞ ¼ 11.

Of course, if our analysis is correct, then these patterns of learning should not be restricted to verbs. Fig. 8 presents the CPP forvarious nouns of different frequencies, showing the highuncertainty scenarios where the differences between early and latelearning should be more pronounced. These plots suggest thatnouns are easier to learn than verbs (we return to this point in Section 3.6), since most of the nouns are learned rapidly, even early on(note the zoomed in plots only go to 1000 utterances here). Nevertheless, nouns observed in the very earliest part of learning do tend
to require more observations to learn than those first observed

O. Abend et al. / Cognition 164 (2017) 116–143

129

Fig. 8. Learning curves showing the CPP for nouns of different frequencies in the (top) 4-distractor and (bottom) 6-distractor settings, as a function of the number ofutterances observed. Right-hand plots zoom in on the earlier stage of learning, as in Fig. 7. The frequencies are: f ðskunkÞ ¼ 2; f ðbirdÞ ¼ 3; f ðbabyÞ ¼ 12;
f ðmanÞ ¼ 29; f ðgirlÞ ¼ 52.

later: in some cases learning is more gradual (as for ‘‘man”); inothers the word may be observed a few times with little or nolearning, before then being learned all at once (as for ‘‘girl”). Figs. 9
and 10 show similar effects for some individual determiners andprepositions.
To summarize, results from looking at learning curves forindividual words suggest that words first observed early inthe learning process are initially learned gradually (or in somecases suddenly but after several occurrences), while those thatfirst appear later on are learned more rapidly, often after onlya single occurrence. These findings, which we explore further
in the following section, suggest that qualitative changes inthe shape of the learning curves can be explained by a single
statistical learning mechanism, which improves as it learns(see also Fazly et al., 2010; Regier, 2005 and references
therein). This single mechanism can also explain the ability
to learn from just one example, a phenomenon we return toin Section 3.7.
3.5. Acceleration of learning and the vocabulary spurt
The learning curves for individual words presented in theprevious section suggested that words encountered early inacquisition are learned more slowly than those first encountered
at a later point. We now show that this finding is not restricted
to the particular words we examined, and we also explore theconsequences for vocabulary size as word learning accelerates
over time.
For the first analysis, we look at transitive verbs. Notice that, on
average, a high-frequency verb will appear first earlier in the corpus
than a low-frequency verb. Thus, if words that first occur late tend
to be learned more rapidly, then on average low-frequency verbs
will be learned in fewer examples, and high-frequency verbs will
require more examples.
To test this hypothesis, we divide verbs into two bins: highfrequency (occurs more than 10 times in the corpus) or lowfrequency (occurs 10 or fewer times). For each bin, we computethe average CPP for all verbs after seeing n instances of that verb
type.15 We plot this average CPP as a function of n in Fig. 11, which
indeed shows the predicted trend.
Our second analysis, which focuses on nouns, is more directlycomparable to observations from real children. Specifically, after
an initial period in which the vocabulary of the child is restricted
to no more than a few words, a sudden increase in the production
vocabulary size is observed, usually around the age of 18–
24 months (Reznick & Goldfield, 1992). The trends we have alreadyobserved in individual words, when taken together over the whole
vocabulary, should yield exactly this kind of accelerated learning.

15
Many verbs are ambiguous between multiple logical forms. We therefore define averb type according to its mv rather than its written form.

130

O. Abend et al. / Cognition 164 (2017) 116–143

Fig. 9. Learning curves showing the CPP for determiners of different frequencies in the (top) 4-distractor and (bottom) 6-distractor settings, as a function of the number ofutterances observed. Right-hand plots zoom in on the earlier stage of learning, as in Fig. 7. The frequencies are: f ðhisÞ ¼ 17; f ðanyÞ ¼ 9; f ðaÞ ¼ 351; f ðmoreÞ ¼ 23;
f ðtheÞ ¼ 321.

Fig. 12 confirms this prediction (see Fazly et al., 2010 for a
related finding). We plot an estimate of the size of the noun production vocabulary as a function of the number of observed
utterances. The noun vocabulary size is defined as the numberof nouns whose CPP exceeds a threshold of 0.8. We present
results for the 6 distractors setting in which the trends can beseen most clearly, as the initial learning stage is longer. Results
show that after an initial period (up to about 200 utterances)in which the vocabulary size increases slowly, the slope of thecurve changes and nouns are learned at a faster rate. This could
be partly due to syntactic bootstrapping from knowledge about
determiner and preposition ordering, which is acquired very
early (as shown in Figs. 5 and 6). It is likely also due to the factthat as more nouns are learned, any sentence containing thosenouns has fewer other unknown words, so overall uncertainty
is reduced. In any case, the results suggest that no special mechanism or change in processing is needed to explain the vocabulary spurt, it simply arises from the nature of the statisticallearner.

acquired for some children (Gentner, 1982; Tomasello,
1992:210). Fig. 13 shows that our learner shows a similar trend,
accumulating nouns faster than transitive verbs, but learning someverbs right from the start.16
This observed bias towards nouns cannot be ascribed to a difference in the conceptual complexity of nouns and verbs, or a difference in their level of concreteness (see Snedeker & Gleitman,
2004 for a review of proposals based on these ideas), as ourmodel does not distinguish nouns and verbs on any suchdimensions.
Moreover, the larger noun vocabulary in our model doesn’t simply reflect the input frequencies of nouns and verbs: Fig. 14 presents the ratio of verb types to noun types in the input, as wellas in the production vocabulary for the different distractor settings.
In the more realistic 2-, 4- and 6-distractor settings, the verb:noun
ratio starts off lower than that in the input, and only later converges to the input ratio. That is, early on (especially in the first
few hundreds of sentences), verbs are under-represented relative
to their input rate. In the no-distractors setting, the learned rate

3.6. Comparing noun and verb learning

16
There are many different ways to count word types, e.g., deciding whether tolump or split a verb by subcategorization, tense, agreement, mood and othergrammatical systems may result in different counts. Here we focus on transitive
verbs, counting types of all inflections, while counting only singular nouns against thenoun’s vocabulary. Thus, any difference between the noun and transitive verb
vocabulary sizes is a conservative estimate. These methodological issues play a lesserrole when comparing the ratios of the input and learned vocabularies (see below).

It is well-known that nouns predominate over verbs in the earlylexicons of English-learning children, although there is considerable individual and cross-linguistic variation, and some verbs or
predicative categories are among the earliest vocabulary items

O. Abend et al. / Cognition 164 (2017) 116–143

131

Fig. 10. Learning curves showing the CPP for prepositions of different frequencies in the (top) 4-distractor and (bottom) 6-distractor settings, as a function of the number ofutterances observed. Right-hand plots zoom in on the earlier stage of learning, as in Fig. 7. The frequencies are: f ðwithÞ ¼ 53; f ðatÞ ¼ 17; f ðinÞ ¼ 86; f ðaboutÞ ¼ 7; f ðonÞ ¼ 78.

Fig. 11. The average CPP (y-axis) for frequent and infrequent verbs, after observing n examples (x-axis). Because the nth example of a low-frequency verb occurs later intraining (on average), fewer examples are required to reach a given level of proficiency. We use a frequency threshold of 10, leading to 99 infrequent verbs and 23 frequentverbs. We do not plot 7–9 occurrences because only eight of the infrequent verbs have 7 or more occurrences. Results are shown only for the 4- and 6-distractor settings
because in the low-distraction setting, all verbs are learned so quickly that it obscures the effect.

is similar to the input rate.17 This under-representation of verbs inthe early phase of learning is consistent with claims of noun bias
in the literature, i.e., that the the ratio of verbs to nouns is lower
17
The plot appears to show that verbs are over-represented for the first 80
utterances, but as the vocabulary is very small at this point (no more than ten wordsof each category), the difference between the learned and input ratios is likely notstatistically significant.

in child productions than in the child directed speech (see discussionin Gentner & Boroditsky, 2001).
Comparing the plots in Fig. 14 with the learning curves for word
order shown earlier suggests an explanation for the noun bias inour model, namely that: (1) early on, nouns are over-represented
in the vocabulary because the syntactic constructions of nounphrases (involving determiners and prepositions) are learned more

132

O. Abend et al. / Cognition 164 (2017) 116–143

Fig. 12. The size of the production vocabulary of nouns as a function of the numberof utterances seen, for the 6 distractors setting. The production vocabulary isdefined as all nouns with a CPP of at least 0.8.

quickly than those of transitive clauses (chiefly, the category oftransitive verbs), allowing syntactic bootstrapping to start earlier
for nouns and increase the rate of noun learning; (2) as learning
progresses, and once the structure of transitive clauses is learned
too, the rate of verb learning ‘‘catches up” to that of nouns, andthe larger noun vocabulary can be directly ascribed to the differing
frequencies of verbs and nouns in child-directed speech. Indeed,
the point of acquisition of the correct SVO category in the different
settings (Fig. 3) largely matches the later phase of convergence
between the input and learned rate. The no-distractor setting thus
does not display an early under-representation of verbs simply
because the syntax of transitives is learned very early on.
This explanation is indeed consistent with the proposal of
Snedeker and Gleitman (2004) that verbs lag behind nouns
because learning verbs requires greater linguistic (rather than conceptual) development on the part of the child. In particular, theyargue that verb learning requires more knowledge about the surrounding sentential context (including its syntax) so the learnergets additional cues beyond just the non-linguistic situational
context.
Beyond this hypothesis, our model also suggests that the use ofsyntactic cues might kick in earlier for nouns than for verbs.

Fig. 14. A comparison of the ratio of transitive verbs and nouns in the childdirected input, relative to this ratio in the child’s production vocabulary (defined aswords with CPP at least 0.8). Curves are presented for all four distractor settings.
Results indicate that the ratios converge to similar values in all settings, but that inthe 2, 4 and 6 distractor settings verbs are initially under-represented relative totheir presence in the input.

Whether this holds true in infants likely requires further behavioral study, since most behavioral work on syntactic bootstrapping
has focused on verbs (see Fisher, Gertner, Scott, & Yuan, 2010 for areview). Nevertheless, some studies do suggest that infants areable to use syntactic and morphological cues earlier in noun processing than verb processing (see Cauvet et al., 2014 for a review).
It is therefore plausible that syntactic information relevant tonouns is learned and used earlier, and that the resulting bootstrapping effect could further explain the early noun bias in children’s
vocabulary.

3.7. Syntactic bootstrapping for one-shot learning of nonce words
So far, all our analyses have focused on the ‘‘natural” timecourseof learning, studying what the simulated child might know at aparticular moment in time as a result of their particular andidiosyncratic history of exposure to language. Much of the literature on child language learning, however, focuses on what childrenof different ages are able to learn based on controlled exposure to asmall number of examples in a laboratory setting. In this section,

Fig. 13. A comparison of the noun and transitive verb vocabulary sizes in the (a) 4-distractor and (b) 6-distractor settings. The curves present the sizes of the production
vocabularies of nouns and transitive verbs (defined as words with CPP at least 0.8) as a function of the number of observed utterances. Results indicate that nouns account fora larger part of the vocabulary than transitive verbs. The relative vocabulary sizes for the 0- and 2-distractor settings are nearly identical to what is shown, but more words arelearned in the 0-distractor setting.

O. Abend et al. / Cognition 164 (2017) 116–143

we follow suit, by first training our model on the first n examples
from the corpus (to simulate the background knowledge a child ofa particular age might bring to the lab), and then exposing it to asmall number of hand-crafted training examples and studying
what it learns from these.
In particular, we examine the model’s ability to deal with cases
like Fisher et al.’s (Fisher, Hall, Rakowitz, & Gleitman, 1994) chase/flee example, where the same physical situation seems to support
more than one logical form. Fisher et al. (1994) ask how it is thatchildren faced with (made up) examples like the following avoid
the error of making an OVS lexical entry for ‘‘flee” with the meaning chase:(7)‘‘Kitties flee doggies”.
It is important that words like ‘‘flee”, where the instigator of theevent is realized as an object rather than a subject, are relatively
rare. (For example, there are eight occurrences of any form of ‘‘flee”
in the entire English CHILDES corpus—over 17 million tokens—of
which none is transitive.) Therefore, the child will have encountered plenty of normal transitive verbs (where the instigator isthe subject) before encountering examples like (7).
In terms of our model, this means that, by the time (7) is
encountered, the learned prior probability of the instantiated rules
for combining transitive SVO verbs with their object and subject
will be substantially greater than the priors for OVS verbs. This willlead to a much higher probability for the correct meaning and SVO
syntax for ‘‘flee” than for an OVS lexical item with the same meaning as ‘‘chase”. Of course, the model also entertains other possibilities, such as that ‘‘flee” means cats, that ‘‘kitties” means flee, etc.
However, assuming that ‘‘kitties” and ‘‘doggies” have been encountered relatively often, so that ‘‘flee” is the only unfamiliar word in
the sentence, then the model will assign a very low probability tothese additional spurious hypotheses.
In fact, we will see that after training, the model can determine
the correct meaning for a transitive verb even in the face of significantly greater ambiguity, when the meanings of other words inthe sentence are not known, because the learned prior over the
OVS syntactic category, which is not attested elsewhere in the data,
becomes exceedingly low.
Gleitman and colleagues have described the process by which
the child resolves this type of contextual ambiguity as ‘‘syntactic
bootstrapping”, meaning that lexical acquisition is guided by thechild’s knowledge of the language-specific grammar, as opposedto the semantics (Gleitman, 1990; Gleitman, Cassidy, Nappa,
Papafragou, & Trueswell, 2005). However, in present terms such133

an influence on learning simply emerges from the statistical model
required for semantic bootstrapping. We will return to this point in
the General Discussion.

3.7.1. Nonce verbs
Our first simulation is inspired by the experiments of Gertner,
Fisher, and Eisengart (2006), where children at the age of 21 and25 months were presented with two pictures, one in which a girlis performing some action over a boy, and one in which it is theother way around. The pictures were paired with a description,either ‘‘the boy gorped the girl” or ‘‘the girl gorped the boy”. Results
showed a disposition of the children to select the interpretation(that is, to look at the picture) in which the actor coincided withthe subject rather than the object of the sentence.
Here, we examine the model’s ability to correctly interpret thenonce verb ‘‘dax” in the sentence ‘‘the man daxed the baby”, whenpresented with two possible meaning representations: one inwhich the man is performing an action over the baby, and one inwhich the baby is performing an action over the man. In order tocorrectly infer the meaning of ‘‘daxed” (analogous to Fisher
et al.’s ‘‘flee” and Gertner et al.’s ‘‘gorped”) the model has to relyon its learned prior to favor SVO over OVS interpretations, as thetwo interpretations are otherwise symmetrical. Results show themodel quickly learns to favor the correct SVO interpretation for‘‘daxed”.
Specifically, we first train the model on the first n examples of
the CHILDES corpus, in intervals of 100 examples (4 and 6 distractors settings), and then present the model with one more sentence – literally, the string ‘‘the man daxed the baby” – which ispaired with both the SVO- and OVS-supporting logical forms.
(Intuitively, the SVO-supporting logical form involving the man
daxing the baby is ‘correct’, while the OVS-supporting logicalform involving the baby daxing the man is a ‘distractor’, but themodel is simply presented both as two a priori equally likely possibilities.) We then query the model’s state to compute several
predictions.
First, we ask which of the two logical forms the model assigns ahigher posterior probability, and find that from n ¼ 200 examples
and onwards the correct SVO interpretation is always preferred.
Next, we use the strict CPP statistic to assess the model’s successat learning the nonce verb ‘‘daxed”: kxky.daxed(y, x) based on thissingle training example (Fig. 15) and find that it is quite successful
despite the syntactic complexity of the surrounding sentence.

Fig. 15. Syntactic bootstrapping: The model’s CPP for the unknown word ‘‘daxed” (y-axis) after training on n utterances from the corpus (x-axis) followed by a single
presentation of the sentence ‘‘The man daxed the baby” with SVO/OVS ambiguous semantics. During training only, either four (a) or six (b) distractors are included along witheach correct logical form.

134

O. Abend et al. / Cognition 164 (2017) 116–143

Fig. 16. Syntactic bootstrapping: The model’s CPP for the word ‘‘daxed” (y-axis) after training on n utterances from the corpus (x-axis) plus one exposure to ‘‘Jacky daxed
Jacob”, where all three words are unknown. During training only, either four (a) or six (b) distractors are included along with each correct logical form. In both cases, thecorrect interpretation of the test sentence is the most likely one upon convergence.

3.7.2. Multiple novel words
The above simulation examined the model’s ability to perform
syntactic bootstrapping to learn an unknown verb in the presenceof uncertainty between an SVO and an OVS interpretation. We next
examine the model’s performance in a simulated experiment where
the utterance is ambiguous between all possible word orders. We
find that also in this setting, the model is able to determine the correct SVO order, due to its high (learned) prior over rules instantiatedby the SVO category as opposed to the alternatives.
Concretely, in this simulation the model is trained on n examples from the corpus, and then receives the single utterance ‘‘Jacky
daxed Jacob”, where ‘‘Jacky”, ‘‘Jacob” and ‘‘daxed” are all unknown
words, paired with the logical form daxed(jacky, jacob). This corresponds to a situation where the child is exposed to a scene inwhich some hitherto unnamed person performs an unusual action
on another unnamed person. A priori, there are many possible
interpretations that are consistent with this utterance; for example, in a VSO language we would expect a child to interpret ‘‘Jacky”
as a verb meaning daxed(x, y), and ‘‘daxed” as a noun meaning
jacky, and if we exposed our model to this example alone without
any other training data, then it would indeed assign this interpretation just as high a probability as the intended SVO interpretation.
What this simulated experiment tests, therefore, is the model’sability to make higher-order syntactic generalizations from theEnglish-based training corpus, and apply them to this novel
utterance.
First, we ask which derivation tree for ‘‘Jacky daxed Jacob”:
daxed(jacky, jacob) is assigned the highest probability, and find thatthe correct SVO derivation is preferred in all cases where n P 500
for the 4 distractor case and n P 800 for the 6 distractor case. We
further examine the model’s ability to assign ‘‘daxed” the correct
SVO interpretation by plotting the CPP of the word ‘‘daxed” after
training on n training examples, followed by the dax example.
Fig. 16(a) and (b) presents this probability for the 4 and 6 distractor
settings respectively. These results demonstrate the model’s ability
to correctly interpret ‘‘daxed” (and consequently the sentence) inthis more ambiguous setting after training on a single example ofthe verb, proving the model’s ability to learn and apply syntactic
generalizations even in the presence of propositional uncertainty.
3.7.3. Nonce nouns and prepositions
Finally, we examine the cases of nouns and prepositions, confirming the model’s ability to perform syntactic bootstrapping inthese categories as well. We take inspiration from Fisher, Klingler,and Song (2006), who demonstrated 2-year old’s disposition tointerpret ‘‘corp” in ‘‘this is a corp” as a noun (and therefore assign
it a noun-compatible meaning), while interpreting ‘‘acorp” in ‘‘thisis acorp the baby” as a preposition (with a preposition-like meaning). In our case, we feed the model with either of the two utterances: ‘‘this is the corp” and ‘‘this is corp Mommy”, paired withtwo possible logical forms: one where ‘‘corp” is compatible with
the noun category and another where ‘‘corp” is compatible withthe preposition category.18 The model assigns ‘‘corp” a noun interpretation in the first example, and a preposition interpretation in
the second. Concretely, as before the model is trained on n examples
in the 4 and 6 distractors settings, and then given a single instance ofeither of the ‘‘corp” examples, paired with the two possible logicalforms. The steep learning curves of this simulation are given in
Fig. 17, indicating that once again the model is able to use its knowledge of syntax to correctly disambiguate between the meanings, justas the children in the experiment did.

4. General discussion
We have presented an incremental model of language acquisition that learns a probabilistic CCG grammar from utterances
paired with one or more potential meanings. The learning model
assumes no knowledge specific to the target language, but does
assume that the learner has access to a universal functional mapping from syntactic to semantic types (Klein & Sag, 1985), as well
as a Bayesian model favoring grammars with heavy reuse of existing rules and lexical types. It is one of only a few computationalmodels that learn syntax and semantics concurrently, and the only
one we know of that both (a) learns from a naturalistic corpus and(b) is evaluated on a wide range of phenomena from the acquisition literature. Moreover, the model is fully generative, meaningthat it can be used for both parsing (understanding) and generation, and that probability distributions over subsets of the variables(e.g., individual lexical items) are well-defined—a property we usedin many of our analyses. Like some very recent proposals(Beekhuizen, 2015; Jones, 2015) but unlike earlier work (Buttery,
2006; Gibson & Wexler, 1994; Sakas & Fodor, 2001; Siskind,
1992; Villavicencio, 2002; Yang, 2002), we also evaluate our learner by parsing sentences onto their meanings, rather than justexamining the learner’s ability to set a small number of predefinedsyntactic parameters.
18
The slight modifications of Fisher et al. (2006)’s examples were done in order toarrive at a pair of utterances of the same length, in which the non-nonce words arelearned very early on.

O. Abend et al. / Cognition 164 (2017) 116–143

135

Fig. 17. The correct production probability (y-axis) for the unknown word ‘‘corp” in the sentences ‘‘This is corp Mommy”, where it should be interpreted as a preposition (aand b), and ‘‘This is the corp” (c and d), where it should be interpreted as a noun, as a function of the number of training examples seen (x-axis). The model is trained in the 4
and 6 distractors settings.

Together, these properties of our model and evaluation allow us
to investigate the relationship between word learning and syntactic acquisition in a more realistic setting than previous work.
Results from our simulations suggest that the kind of probabilistic
semantic bootstrapping model proposed here is sufficient to
explain a range of phenomena from the acquisition literature:acceleration of word learning, including one-shot learning of many
items later in acquisition; the predominance of nouns over verbs inearly acquisition; and the ability to disambiguate word meanings
using syntax. All of these behaviors emerge from the basic assumptions of the model; no additional stipulations or mechanisms are
required. Finally, our work addresses a criticism levelled against
some statistical learners—that their learning curves are too gradual(Thornton & Tesan, 2007). By demonstrating sudden learning ofword order and of word meanings, our model shows that statisticallearners can account for sudden changes in children’s grammars.
In the remainder of the discussion, we expand on a few of thesepoints, highlight some connections between our model and earlier
theories of acquisition, and discuss how our model might generalize to more realistic meaning representations.
4.1. Syntactic bootstrapping revisited
It has long been noted that the onset of syntactically productive
language is accompanied by an explosion of advances in qualitatively different ‘‘operational” cognitive abilities (Vygotsky,

1934/1986). These findings suggest that language has a feedback
effect that facilitates access to difficult concepts, a view that is further supported by the early work of Oléron (1953) and Furth(1961), who demonstrated that deaf children who were linguistically deprived (by being denied access to sign) showed specificcognitive deficits in non-perceptually evident concepts.
This view is also consistent with Gleitman’s (Gleitman, 1990)
proposal that the availability of syntax enables the child to ‘‘syntactically bootstrap” lexical entries for verbs (such as ‘‘promise”)
that may not be situationally evident. However, we have argued
that such syntactic bootstrapping is only possible when the childhas access to a model of the relation between language-specificsyntax and the universal conceptual representation. Our simulations of learning novel transitive verbs, inspired by the chase/flee
ambiguity and related behavioral studies, support this argument
and demonstrate that syntactic bootstrapping need not be postulated as a distinct mechanism, but is emergent from a semantic
bootstrapping approach such as the one proposed here.
In that sense, the so-called ‘‘syntactic” bootstrapping effectmight better be termed ‘‘structural”, since it relies crucially on(and arises directly from) the homomorphism between syntactic
and semantic types. Although the child might eventually learn
the meanings of novel predicates in any case, structural bootstrapping serves to highlight the meaning that is most plausible in lightof the child’s previously acquired grammatical knowledge. Ourmodel operationalizes that previously acquired knowledge as a

136

O. Abend et al. / Cognition 164 (2017) 116–143

probabilistic grammar, where the probabilities learned from previous examples act as the prior distribution when faced with a new
example. When learning from a naturalistic corpus, these priors
become sufficiently peaked that the learner is able to identify thecorrect verb meanings in examples with ambiguous semantic roles,
like ‘‘the man daxed the baby”. More generally, the learner’s accumulating syntactic knowledge (along with knowing more word
meanings) causes an acceleration of learning, which we demonstrated both for verbs and nouns.

4.2. Relationship to non-probabilistic and parameter-setting
approaches
Like other probabilistic approaches to language acquisition, ourproposal avoids many of the stipulations required by nonprobabilistic approaches. In particular, there is no need for a Subset
Principle (Berwick, 1985) or the ordered presentation of unambiguous parametric triggers, both of which appear to present serious problems for the language learner (Angluin, 1980; Becker,
2005; Fodor & Sakas, 2005). Nor does our approach contradict
widely-held assumptions concerning the lack of negative evidence
in the child’s input. The child can progress from the universal
superset grammar to the language-specific target grammar
entirely on the basis of positive evidence. This positive evidence
raises the probability of correct hypotheses at the expense of incorrect ones, including those introduced by error and noise. The only
evidence that the child needs is a reasonable proportion of utterances that are sufficiently short for them to deal with.
Nevertheless, our proposal does share some commonalities
with earlier non-probabilistic approaches. For example, it resembles the proposal of Fodor (1998) as developed in Sakas and Fodor(2001) and Niyogi (2006) in that it treats the acquisition of grammar as arising from parsing with a universal ‘‘supergrammar”,
which includes all possible syntactic structures allowed in any
language that are compatible with the data (Fodor, Bever, &Garrett, 1974:475). However, our learner uses a different algorithm to converge on the target grammar. Rather than learning
rules in an all or none fashion on the basis of parametricallyunambiguous sentences, it uses all processable utterances toadjust the probabilities of all elements of the grammar for which
there is positive evidence. In this respect, it more closely resembles the proposal of Yang (2002). However it differs from both ineschewing the view that grammar learning is parameter setting,and from the latter in using a Bayesian learning model rather
than Mathematical Learning Theory (Atkinson, Bower, &Crothers, 1965).
We eschew parameter setting because if the parameters are
implicit in the rules or categories themselves, which can belearned directly, neither the child nor the theory need be concerned with parameters other than as a prior distribution overthe seen alternatives. For the child, all-or-none parametersetting would be counterproductive, making it hard to learn the
many languages that have exceptional lexical items or inconsistent settings of parameters across lexical types, as in the English
auxiliary verb system and Italian subject-postposing intransitives.
Of course, languages do tend toward consistent (if violable) values of parameters like headedness across categories for related
semantic types (such as verbs and prepositions). Such consistency
will make learning easier under the present theory, by raising themodel’s probabilities for recurring rules and categories, leading tohigher prior probabilities for those rules and categories when
they are re-encountered in future. It is less clear that representing
this consistency explicitly, rather than implicitly in the model andthe evolving prior, will help the individual child learning its first
language.

4.3. Cognitive plausibility and interpretation of probabilities
Our model is based on similar mathematical principles to other
Bayesian models of language acquisition (e.g., Feldman et al., 2013;
Frank et al., 2009; Goldwater et al., 2009; Griffiths & Tenenbaum,
2005, 2006; Perfors, Tenenbaum, & Wonnacott, 2010; Xu &Tenenbaum, 2007), and shares with them the attractive properties
noted above: the probabilistic framework permits the learner toaccumulate evidence over time and eventually make strong inferences about the language-specific grammar, without the need fornegative evidence and in a way that is robust to noise. However,unlike many Bayesian models, our learner uses an incremental
learning algorithm rather than adopting an ideal observer
approach. We adopted this approach in the interests of cognitiveplausibility and in order to easily examine the progress of learningover time. As a result our learner has no guarantee of optimality,
but is aligned with other recent work investigating cognitively
plausible approximate learning strategies for Bayesian learners(Pearl, Goldwater, & Steyvers, 2010; Sanborn, 2017; Sanborn,
Griffiths, & Navarro, 2010; Shi, Griffiths, Feldman, & Sanborn,2010).
Nevertheless it is worth noting that, despite the incremental
algorithm, some other aspects of our learner are less cognitively
plausible. In particular, to process each input example the learnergenerates all parses consistent with that example. This can requiresignificant memory and computation, especially for longer sentences, and indeed is not incremental at the level of words, as
human sentence processing is generally accepted to be. However,it might be possible to develop further approximations to reducethe computational requirements, for example by computing
parameter updates based on a high probability subset of the possible parses. Another approach might be particle filtering, which has
been used by some other recent models to model memory limitations in incremental sentence processing and other tasks (Levy,
Reali, & Griffiths, 2009; Sanborn et al., 2010; Shi et al., 2010).
Regardless of the learning algorithm, the use of probabilities
raises the question of how exactly a notion like ‘‘the adult grammar” should be understood in this model, where the grammar represents a probability distribution over all hypotheses that the childlearner has ever entertained, albeit one that has converged withmost of the probability mass concentrated in correct lexical entries
and rules of derivation, as opposed to spurious determiners like‘‘doggies” and rules for combining them with nouns like ‘‘more”.
Such a grammar/model could in principle be pruned to eliminate such obvious dead wood. However, the grammar would
remain probabilistic in nature. This conclusion might seem to conflict with traditional theories of competence, but such probabilistic
grammars are now widespread in models of gradience in language
processing and gradient grammaticality (Bresnan & Nikitina, 2003;
Hale, 2001; Jurafsky, 1996; Levy, 2008; Sorace & Keller, 2005).
Some have argued that gradience in competence is distinct fromthat in performance (Crocker & Keller, 2005). Our model’s probabilities closely reflect processing (i.e., frequency of use). However,even if interpreted as a model of competence, it can in principlemodel both gradient and hard judgements, the latter either as
extremely low-probability events, or as structures that are entirely
absent from the probabilistic grammar. Our model would theninstantiate a theory of learning for such gradient grammars, analogous to various learning models proposed for stochastic Optimality Theoretic grammars (e.g., Boersma & Hayes, 2001; Goldwater &Johnson, 2003; Hayes & Wilson, 2008).
4.4. Towards more realistic meaning representation
The meaning representations for child-directed utterance considered in this work have been necessarily simplified in content,

O. Abend et al. / Cognition 164 (2017) 116–143

because the CHILDES annotations that we have to work with are
limited to narrow, propositional literal meaning. However, the content that the child learns from surely also includes aspects of thesocial interaction that the child is engaged in (Tomasello &Farrar, 1986 passim). It is the latter that provides the primary motivation for the child to learn language, particularly in the earliest
stages. Such interpersonal content is lacking in the CHILDES
annotation.
We know of no fully convincing or widely accepted formal
semantic representation for these interpersonal aspects of discourse meaning, much less any such representation that we
believe human annotators could apply consistently to labelled corpora of child directed utterance (Calhoun et al., 2010; Cook &Bildhauer, 2011).
However, we do believe that the knowledge representation
identified by Tomasello can be expressed as compositional semantic formulas, not fundamentally different from the ones we use
here to demonstrate the workings of our learner. Indeed,
Steedman (2014) proposed such a compositional formulation for
one major component of interactional learning, namely information structure.
The term ‘‘information structure” refers to the packaging of thecontent of an utterance along a number of dimensions that havebeen referred to in the literature—not always consistently—as‘‘topic”, ‘‘comment”, ‘‘contrast” and the like, together with markersof speaker and hearer orientation of these meaning components(Bolinger, 1965; Halliday, 1967; Ladd, 1996; Steedman, 2014),
often expressed by prosodical markers such as pitch contour andmetrical alignment (Calhoun, 2010, 2012). Intonational prosody
is used exuberantly in speech by and to children, and we know that
infants are sensitive to the interpersonal significance of intonationfrom a very early age (Fernald, 1993; Fernald et al., 1989).
It is therefore likely that when a child hears her mother pointing
at newly observed dogs and saying ‘‘More doggies!”, the child’s
representation of the utterance includes the information that a
new entity or property is introduced to the common ground, andthat she considers the possibility that this information is conveyed
by the speaker’s intonation contour. Using the intonational notation of Pierrehumbert and Hirschberg (1990) (H⁄ and LL% are a highaccent and low boundary tone respectively) and sub-categorizing
the syntactic categories to correspond to information-structural
distinctions, the derivation might look as follows (þ; q marks apositive polarity rheme or comment, and / marks a complete
phonological phrase):

137

of derivational structure. Consider the child in a similar situation to
the above, who hears example (1) from Section 2.1 ‘‘You like thedoggies!” with the intonation contour discussed by Fisher herself
and by Steedman (1996a)19:‘‘Mum supposes the properties the dogs afford to be common
ground, Mum makes it common ground it’s what I like.”
Fisher points out that the L intermediate phrase boundary that
she observed after the verb makes the intonation structure inconsistent with standard assumptions about English NP-VP surface
constituency. However, CCG embodies a more permissive notionof constituency, allowing for constituents such as ‘‘you like”, consistent with the intonation structure above. As the derivation
requires the use of the forward composition rule, indicated as> B, the intonation contour provides positive evidence for applying
the composition rule to the first two categories, supporting syntactic learning from prosodic cues. Far from being inconsistent withthe syntax of English, the intonation structure is aligned to thenotion of constituent structure that the child will later find essential to the analysis of relative clause structures like the following:(10) [The [doggie [that [you like]S/NP]NnN]N]NP
Thus, the child can acquire the relative construction without
needing to call on a distinct and otherwise unmotivated mechanism of movement. This strong homomorphism between interpretation and syntactic constituency motivates the use of CCG as themodel of grammar in a theory of language acquisition.
It is also worth noting that even the content that is expressed in
our input meaning representations is inevitably more shallow andEnglish-specific than we would like, even though it abstracts awayfrom English-specific word order. For example, comparison withother Germanic languages suggests that the logical form that thechild brings to (9) may be something more like give(pleasure(you),dogs), so that the lexical entry for ‘‘like” of type ðe; ðe; tÞÞ isthe following, exhibiting the same ‘‘quirky” relation between(structural) nominative case and an underlying dative role that Icelandic exhibits morphologically for the corresponding verb:(11) like ‘ ðSnNPÞ=NP : kxky:giv eðpleasureðyÞ; xÞThe semantics of this sentence can be glossed as: ‘‘Mum makes
the property afforded by more dogs common ground”. The detailsof the semantics of information structure go beyond the immediateconcerns of this paper: the example is merely intended to suggestthat interpersonal aspects of meaning can be treated as formally as
more standard content.
Intonation structure does not always align with traditional syntactic structure even in adult dialog, and the two diverge evenmore in child-directed and child-originated speech (Fisher &Tokura, 1996; Gerken, 1996; Gerken, Jusczyk, & Mandel, 1994).
However, in CCG, intonation structure is united with a freer notion

However, dealing with these further issues is beyond the scopeof the present investigation, because of the small amount of data
we have to work with, and the restricted nature of the annotation
available to us.
19This derivation crucially involves type-raising of the subject You. Type raising
simply exchanges the roles of function and argument, making the subject apply to thepredicate to give the same result. Although, in the interests of simplification, we haveglossed over the point in the preceding discussion of the learner, all NPs are always
type-raised in this sense by a lexical process analogous to morphological case, or what
the linguists call ‘‘structural” case, so the object is also type-raised. Thus, this example
involves the CCG equivalent of nominative and accusative case. The responsibility for
deciding which structural case is relevant lies with the parsing model, rather than thegrammar.

138

O. Abend et al. / Cognition 164 (2017) 116–143

5. Conclusion
This paper has presented, to our knowledge, the first model ofchild language acquisition that jointly learns both word meanings
and syntax and is evaluated on naturalistic child-directed sentences paired with structured representations of their meaning.
We have demonstrated that the model reproduces several important characteristics of child language acquisition, including rapid
learning of word order, an acceleration in vocabulary learning,
one-shot learning of novel words, and syntactic bootstrapping
effects.
Taken together, these results support our basic claim that syntax is learned by semantic bootstrapping from contextually available semantic interpretations using a statistical learning
mechanism that operates over a universally available set of grammatical possibilities. This learning mechanism gradually shifts
probability mass towards frequent candidates, allowing the childto rapidly acquire an (approximately) correct grammar even in
the face of competing ambiguous alternative meanings. Altogether,
the model offers a unified account of several behavioral phenomena, demonstrating in particular that both syntactic bootstrapping
effects and changes in the pace of learning emerge naturally fromthe kind of statistical semantic bootstrapping model presented
here—no additional mechanisms are needed.

We thank Julia Hockenmaier and Mark Johnson for guidance,and Inbal Arnon, Jennifer Culbertson, and Ida Szubert for their
feedback on a draft of this article.
Appendix A. The probabilistic model
Recall the probabilistic model, defining a family of distributions
over ðs; m; tÞ triplets consisting of an utterance, meaning representations for the leaves, and a derivation tree. The model is an
instance of a Probabilistic Context-free Grammar (PCFG), anddecomposes over the derivation steps (or expansions). Specifically,
let w1 ; . . . ; wn be the words in the sentence s, m1 ; . . . ; mn be their
meaning representations and c1 ; . . . ; cn be their syntactic categories(the syntactic categories at the leaves of t). The probability of an
ðs; m; tÞ triplet is:YPðc j aÞ 

a!c2t

n
n
Y
Y
Pðmi j ci Þ  Pðwi j mi Þ
i¼1

na!c þ asyn Ha
na þ asynFirst we define the base distribution of syntactic expansions.
We begin by defining an auxiliary distribution over the space ofsyntactic categories, i.e., the set of all categories that can be definedusing brackets and slashes over the set of atomic categories. For a
category c, we define nslashes as the number of slash symbols in c(e.g., nslashes of an atomic category is 0 and nslashes ðSnNP=NPÞ is 2).
Then:

ð10Þ

ð11Þ

where na!c is the number of times the expansion a ! c has beenused previously and na is the number of times any expansion
headed by a has been used (in fact, since we never actually observe
any parses, the algorithm we describe in Appendix C defines na!cand na as the expectations of these numbers). The model has two

ð12Þ

While this does not define a proper distribution over all syntactic categories of any length (as Rc Prcats ðcÞ is unbounded), it can benormalized if we assume the vocabulary and formula length arebounded, resulting in a well-defined distribution.
The possible expansions from a syntactic category X depend on
its identity. We first define nop ðXÞ as the number of possible CCG
inverted operators that can apply to X. Any category X can expand
using right and left inverted application, resulting in ðX=Y; YÞ andðY; XnYÞ. If X is complex (e.g., X ¼ X1 =X2 ), it can also be expanded
through inverted composition (e.g., to X1 =Y and Y=X2 ). In general,
if X is complex and takes its first k arguments to its right (left), it
can be expanded through right (left) inverted composition oforders 1; . . . ; k. nop can be defined accordingly. For each of theallowable operators, the expansion is fully determined by the identity of the introduced category (Y in the examples above). In addition, a category may always be expanded into a lexical item bygenerating the symbol XLEX . Ha for a syntactic category a is thendefined as:(i¼1

Defining the model consists of defining the distribution of thedifferent expansions: the distribution Pðc j aÞ over each syntactic
expansion c (consisting of one or more nonterminal syntactic categories) given its parent nonterminal category a; the distributionPðmi j ci Þ over each leaf meaning representation given its (terminal)
syntactic category; and the distribution Pðwi j mi Þ over each word
given its meaning representation. We refer to all these cases collectively as expansions of the parent category (a; ci or mi ). The distributions of the expansions are defined using Dirichlet Processes (DPs).
A DP can serve as the prior over a discrete distribution with infinite
support, and (as noted in Section 2.3) yields the following posterior
distribution over expansions:

Pðc j aÞ ¼

A.1. Syntactic expansions

Prcats ðcÞ / 0:2nslashes ðcÞþ1

Acknowledgements

Pðs; m; tÞ ¼

hyperparameters: the base distribution Ha and concentration
parameter aa , discussed below. We use here the notation for syntactic expansions, but the same equation applies to the expansion of acategory into meaning representation c ! m or meaning representation into wordform m ! w.
A Dirichlet Process is therefore defined by the concentration
parameter and base distribution (which are stipulated in thiswork), and the sufficient statistics na!b for every a; b, which are
estimated from the training data. As only a finite number of ða; bÞpairs are non-zero at any given point of the inference procedure,
we store the pseudo-counts for these pairs explicitly. The following
subsections define the base distributions and concentration
parameters used in each part of the model.

Ha ðbÞ /0:5  CðbÞ b ¼ aLEX
Prcats ðYÞ2nopb is a syntactic expansion; introducing Yð13ÞCðbÞ equals 1 if nslashes ðbÞ < 2, equals 43 if b’s innermost slashes are in
the same direction (both arguments to the right or both to the left,
as in X=Y=Z), and equals 23 otherwise (innermost slashes in different
directions, as in X=YnZ). This term corrects the base distribution to
be uniform over the six possible constituent orders for a transitive
verb after the exclusion of the two cases where the verb attaches
first to the first semantic argument (see Section 2.2).
A.2. Meaning representations
The distribution of the expansion of leaf categories into meaning representations Pðmi j ci Þ is done in two steps. First, ci generates
a shell-meaning representation, which does not contain any constants but instead has (typed) placeholders. Another generation
step is then used to expand the shells into fully-fledged meaning
representations. For instance, to generate the meaning representa-

139

O. Abend et al. / Cognition 164 (2017) 116–143

tion mi ¼ kx:cookieðxÞ, the corresponding leaf ci first generatesshmsh
i ¼ kx:PðxÞ, and then generates mi given mi .
The reason for performing this two-step derivation is to allow
the model to learn a disposition towards sub-classes of logicalforms. For instance, assume the learner is exposed to a VerbSubject-Object language, and correctly learns that the category
S=NP=NP is more likely than the other transitive categories. Now
assume the learner faces a transitive clause, in which none of thewords are known w1 w2 w3 : pðarg 1 ; arg 2 Þ. The model will correctly
assign high probability to analyses where w1 corresponds to p,
but without the additional ci ! msh will be unable to determine
whether w2 corresponds to arg 1 and w3 to arg 2 or the other way
around. Recall that the VSO and VOS derivations differ only inthe meaning representation assigned to w1 (Section 2.2). Specifically, VSO involves the expansion kxky:pðx; yÞ ! w1 and VOS theexpansion kykx:pðx; yÞ ! w1 . Both of these derivations have notbeen previously observed and are symmetrical. A single derivation
step would therefore assign them equal probabilities, regardless ofwhether other learned transitive verbs have VSO or VOS lexical
entries.
Using the intermediate shell generation step addresses this problem. If the observed data better attests for VSO than VOS verbs, thederivation S=NP=NP ! kxky:PLACE HOLDERðx; yÞ would gain ahigher probability mass than S=NP=NP ! kykx:PLACE HOLDERðx; yÞ.
Consequently, the VSO interpretation for w1 would be preferred overthe VOS.
Both derivation steps ci ! msh
and mshii ! mi are modelled as20
DPs. The base distribution in both cases is similarly defined21:

Hsem ðmÞ / e#symþ2#v ars

ð14Þ

where #sym is the number of constants, logical operators and quantifiers appearing in m, and #v ars is the number of variable types(not instances) appearing in m. Again this is an improper distribution, but can be normalized if we assume the vocabulary and formula length are bounded.
The inclusion of shell logical forms in the derivation processserves an additional purpose when inferring meaning representations for sentences s that contain unknown words. While accurateinference is impossible in this case, as the model has no information what logical constants correspond to the unknown words, it
is able to infer the most likely logical form, where for unknown
words wi ; msh is inferred instead of mi . This may result in a meaning
representation for the sentence that contains place-holders insteadof some of the logical constants, but is otherwise accurate.

A.4. Concentration parameters
The concentration parameters we use are asyn ¼ 1 for the syntactic expansions, ash ¼ 1000 for the expansions of syntactic categories to shells, am ¼ 500 for the expansions of shells to meaning
representations, and aw ¼ 1 for the expansion of meaning representations to words.
Appendix B. The space of possible derivations
The set of allowed parses t is defined by the function T,
mapping pairs of strings and meaning representations ðs; mÞ into
a set of possible derivations. Here we review the splitting procedure of Kwiatkowski et al. (2010) that is used to generate CCG
lexical items and describe how it is used by T to create apacked chart representation of all parses t that are consistent
with ðs; mÞ.
The splitting procedure takes as input a CCG category X : h,such as NP : aðx; cookieðxÞÞ, and returns a set of category splits.
Each category split is a pair of CCG categories ðCl : ml ; Cr : mr Þ
(ordered by their linear order) that can be recombined to giveX : h using one of the CCG combinators in Section 2.2. The CCG
category splitting procedure has two parts: logical splitting of
the category semantics h; and syntactic splitting of the syntactic
category X. Each logical split of h is a pair of lambda expressions
ðf ; gÞ in the following set:fðf ; gÞ j h ¼ f ðgÞ _ h ¼ kx:f ðgðxÞÞg;ð16Þ

which means that f and g can be recombined using either function
application or function composition to give the original lambda
expression h. A number of further linguistically motivated constraints on possible splits apply, such as an ‘‘Across the Board” constraint that says that abstraction over a term A must apply to all
instances of A, and an ‘‘A over A” constraint that says you only consider the topmost term of type A for abstraction, and not any termsof the same type A that it may contain (cf. Ross, 1986). These are similar to the constraints described in Kwiatkowski (2012, 116–118). An
example split of the lambda expression h ¼ aðx; cookieðxÞÞ is the pairðky:aðx; yðxÞÞ; kx:cookieðxÞÞ;

ð17Þ

where ky:aðx; yðxÞÞ applied to kx:cookieðxÞ returns the original
expression aðx; cookieðxÞÞ.
If f and g can be recombined to give h with function application,and X is the syntactic category of h, then X is split by a reversal of
the CCG application combinators in Section 2.2, yielding the following set of possible categories for f and g:

A.3. Wordforms

fðX=Y: f

The base distribution of the generation of words from logicaltypes mi ! wi is defined as follows:

where X : h and h = f(g). Similarly, if f and g can be recombined togive h with function composition and if X = X1 = X2 , then X is split
by a reversal of the CCG composition combinator, yielding:

Hword ðwÞ ¼ 0:002jwj

fðX1 =Y: f

ð15Þ

where jwj is the number of characters in w.
20
We use the same DP for ci ’s that only differ in the slash direction. For instance,
ci ¼ S=NP and ci ¼ SnNP share a distribution of Pð j ci Þ. This was done to prevent anartefact where unknown meaning representations generated from an infrequent
category receive disproportionally large probability mass relative to their probability
for being generated from a frequent category. This modification introduces anadditional simplifying independence assumption (namely that the distribution of
meaning representations does not depend on the directionality of the corresponding
leaf syntactic category), which we intend to review in future work.
21
Much of the space over which the base distribution is defined cannot be attested
for due to type constraints. For instance, where the leaf category is of type ci ¼ NP, theresulting meaning representation must be of type e. Nevertheless, we use this basedistribution uniformly for its simplicity.

Y: gÞ; ðY: g

XnY: f Þg

Y=X2: gÞg

ð18Þ

ð19Þ

where X : h and h = kx.f(g(x)). Analogous inverted combinators
apply for the case of X ¼ X1 nX2 and higher-order inverted composition combinators.
The categories that are introduced as a result of a split (Y) arelabelled via a functional mapping cat from semantic type T to syntactic category:89
if T 2 Fig:D1 >>< AtomicðTÞ
=catðTÞ ¼ catðT 1 Þ=catðT 2 Þ if T ¼ hT 1 ; T 2 i>>:;catðT 1 ÞncatðT 2 Þ if T ¼ hT 1 ; T 2 i

which uses the Atomic function illustrated in Fig. D1 to map
semantic-type to basic CCG syntactic category.

140

O. Abend et al. / Cognition 164 (2017) 116–143

Fig. D1. Atomic syntactic categories and their corresponding semantic types.

As an example, the logical split in (17) supports two CCG category splits, one for each of the CCG application rules.

ðNP=N : ky:aðx; yðxÞÞ; N : kx:cookieðxÞÞ

ð20Þ

ðN : kx:cookieðxÞ; NPnN : ky:aðx; yðxÞÞÞ

ð21Þ

The parse generation algorithm T uses the function split togenerate all CCG category pairs that are an allowed split of an input
category X : h:fðCl : ml ; Cr : mr Þg ¼ splitðX : hÞ;ð22Þ

and then packs a chart representation of t in a top-down fashion
starting with a single cell entry Cm : m for the top node shared by
all parses t. T cycles over all cell entries in increasingly small spansand populates the chart with their splits. For any cell entry X : hspanning more than one word T generates a set of pairs representing the splits of X : h. For each split ðCl : ml ; Cr : mr Þ and every binary
partition ðwi:k ; wk:j Þ of the word-span T creates two new cell entries
in the chart: ðCl : ml Þi:k and ðCr : mr Þk:j .
Algorithm 1. Generating t with T.
Input: Sentence ½w1 ; . . . ; wn , top node Cm : m
Output: Packed parse chart Ch containing t
Ch ¼ ½½fg1 ; . . . ; fgn 1 ; . . . ; ½fg1 ; . . . ; fgn n 
Ch½1½n  1 ¼ Cm : m
for i ¼ n; . . . ; 2; j ¼ 1 . . . ðn  iÞ þ 1 do
for X : h 2 Ch½j½i dofor ðCl : ml ; Cr : mr Þ 2 splitðX : hÞ dofor k ¼ 1; . . . ; i  1 do
Ch½j½k
Cl : ml
Ch½j þ k½i  k
Cr : mr
Algorithm 1 shows how the learner uses T to generate a packed
chart representation of t in the chart Ch. The function T massively
overgenerates parses for any given natural language. The probabilistic parsing model (see Appendix A) is used to choose the bestparse from the overgenerated set.

Appendix C. The learning algorithm
The parameters of the model are estimated incrementally basedon one example at a time. Each example is a pair ðs; mÞ of an input
sentence s and a set of possible meanings m. The algorithm estimates the posterior distribution Pðt; H j s; mÞ: the joint distribution
over possible derivations t of s and the model parameters H.
Since estimating this distribution exactly is intractable, we use
a mean field approximation, considering a simpler family of
distributions Q ðt; HÞ such that Q decomposes as follows:Q ðt; HÞ ¼ Q ðtÞQ ðHÞ

We may now approximate D ¼ Pðt; H j s; mÞ by finding the distribution Q ðt; HÞ that minimizes the KL divergence from D using the
Variational Bayes EM algorithm (VBEM). We adapt existing techniques for online VBEM (Beal, 2003; Hoffman, Blei, & Bach, 2010;
Sato, 2001).
The algorithm iteratively performs two steps. Given an
instance ðs; mÞ, the first step (‘‘oVBE-step”) creates a packed forest representation t of the space of possible derivations thatyield any of the observed sentence-meaning pairs (see Appendix
B for details). This packed representation is used to estimate
the expected number of times the expansion a ! c appears in
the derivation space for the current example: Et ½a ! c. (Asabove, we use the notation for syntactic expansions, but thealgorithm also considers the expansion of a category into meaning representation c ! m or meaning representation into wordform m ! w.) The second step (‘‘oVBM-step”) then computes
the estimated number of times that a ! c appeared in any
derivation of any of the observed examples. These estimatescorrespond to the pseudo-counts na!c that define the Dirichlet
Process defining the distribution over the expansions of a. The
computation is done by extrapolating the previous value ofna!c with Et ½a ! c. In short, the oVBE step estimates the distribution over latent derivations t given a (sentence, possible
meanings) pair, while the oVBM step estimates the parametersof the model, represented as pseudo-counts na!c , given that
distribution.
The full algorithm is presented in Algorithm 2. For each s andpossible meaning representation m0 2 m it uses the function T
to generate a set of consistent parses t0 .
In the parameter update step, the training algorithm updates
the pseudo-counts associated with each of the expansions a ! cthat have been observed in any of the derivations generated forthis example.
We follow Hoffman et al. (2010) in using an online update
schedule that knows, prior to training, the number of training
instances that will be seen N. This is a small contravention of theconditions required for the algorithm to be truly online. In practice,
the value of N is used by the algorithm to weight the effects of theprior against those of the observed data and need not exactly
match the amount of data to be seen. We further follow Hoffman
et al. (2010) in introducing a learning rate gi ¼ ð50 þ iÞ
, wherei is the iteration number. The more gradual learning rate, discussed
0:8

in Section 3.3, is gi ¼ ð50 þ iÞ

0:6 22.

22
For brevity, some of the figures in Section 3.3 are only presented for the 4 and 6
distractor settings. For transitives with the more gradual rate, the correct SVO
category is only favored by a small margin over the alternatives, which is reflected in
the CPP for individual transitives and in the CPP of the nonce transitive after a single
exposure: they are both low in absolute terms. In all other experiments and settings,
the model behaves similarly as in the 4 and 6 distractor settings.

O. Abend et al. / Cognition 164 (2017) 116–143

Algorithm 2. Learning the lexicon and the model parameters.
Input: Corpus D ¼ fðsi ; mi Þ j i ¼ 1; . . . ; Ng, Function T,
Semantics to syntactic category mapping cat.
Output: Model Parameters fna!c g
for i ¼ 1; . . . ; N doti ¼ fg
for m0 2 mi do
Cm0 ¼ catðm0 Þ
ti ¼ ti [ Tðsi ; Cm0 : m0 Þ
for a ! c 2 ti do
oVBE-step: Compute Eti ½a ! coVBM-step: Update na!c ð1  gi Þna!c þ gi  N  Eti ½a ! bAppendix D. Examples of sentence-meaning pairs used in thesimulation experiments
Below are examples of sentence-meaning pairs taken from theconverted dependency-annotated Eve corpus used in the simulation experiments. Predicate symbols and constants are written in
the form POS j af , where POS is the symbol’s part of speech, and f
are additional features, where applicable (often inflectional features). However, for the purposes of this work, all symbols are
viewed as non-decomposable. The parts of speech determine thesemantic types, which are here omitted for brevity.
you go: : kev:v j goðpro j you; evÞyou get a fly: : kev:v j getðpro j you; det j aðe; n j flyðeÞÞ; evÞ
who are you calling ? :ke:kev:aux j bePRESENT ðpart j callPROGRESSIVE ðpro j you; e; evÞ; evÞyou read about the choochoo: :kev:v j readZERO ðpro j you; evÞ^prep j aboutðdet j theðe; n j choochooðeÞÞ; evÞReferences
Abend, O., Reichart, R., & Rappoport, A. (2010). Improved unsupervised POS
induction through prototype discovery. In Acl (pp. 1298–1307).
Alishahi, A., & Chrupała, G. (2012). Concurrent acquisition of word meaning andlexical categories. In Proceedings of the 2012 joint conference on empiricalmethods in natural language processing and computational natural language
learning (pp. 643–654). Jeju Island.
Alishahi, A., Fazly, A., & Stevenson, S. (2008). Fast mapping in word learning: What
probabilities tell us. In Proceedings of the twelfth conference on computational
natural language learning (pp. 57–64).
Alishahi, A., & Stevenson, S. (2008). A computational model of early argument
structure acquisition. Cognitive Science, 32, 789–834.
Alishahi, A., & Stevenson, S. (2010). A computational model of learning semanticroles from child-directed language. Language and Cognitive Processes, 25(1),50–93.
Allen, J., & Seidenberg, M. S. (1999). The emergence of grammaticality inconnectionist networks. The Emergence of Language, 115–151.
Ambati, B. R., Deoskar, T., & Steedman, M. (2017). Hindi CCGbank: A CCG treebank
from the Hindi dependency treebank. Language Resources and Evaluation, 1–34.
Ambridge, B., Pine, J., & Lieven, E. (2014). Child language acquisition: Why universal
grammar doesn’t help. Language, 90, e53–e90.
Angluin, D. (1980). Inductive inference of formal languages from positive data.
Information and Control, 45, 117–135.
Artzi, Y., Das, D., & Petrov, S. (2014). Learning compact lexicons for CCG semantic
parsing. In Proceedings of the 2014 conference on empirical methods in natural
language processing (pp. 1273–1283).
Atkinson, R., Bower, G., & Crothers, E. (1965). Introduction to mathematical learningtheory. Wiley.
Auli, M., & Lopez, A. (2011). A comparison of loopy belief propagation and dual
decomposition for integrated CCG supertagging and parsing. In Proceedings ofthe 49th annual meeting of the Association for Computational Linguistics: Human
language technologies (pp. 470–480). Portland, OR: ACL.
Baldridge, J. (2002). Lexically specified derivational control in Combinatory Categorial
Grammar (Unpublished doctoral dissertation). University of Edinburgh.

141

Barak, L., Fazly, A., & Stevenson, S. (2013). Modeling the emergence of an exemplar
verb in construction learning. In Proceedings of the 35th annual conference of the
Cognitive Science Society. Berlin.
Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference(Unpublished doctoral dissertation). University of London.
Becker, M. (2005). Raising, control, and the subset principle. In Proceedings of the24th West Coast conference on formal linguistics (pp. 52–60). Somerville, MA:
Cascadilla Proceedings Project.
Beekhuizen, B. (2015). Constructions emerging: A usage-based model of the acquisitionof grammar (Unpublished doctoral dissertation). Leiden University.
Beekhuizen, B., Bod, R., Fazly, A., Stevenson, S., & Verhagen, A. (2014). A usage-basedmodel of early grammatical development. In Proceedings of the ACL workshop on
cognitive modeling and computational linguistics. Baltimore, MD.
Berwick, R. (1985). The acquisition of syntactic knowledge. Cambridge, MA: MIT Press.
Boersma, P., & Hayes, B. (2001). Empirical tests of the gradual learning algorithm.
Linguistic Inquiry, 32(1), 45–86.
Bolinger, D. (1965). Forms of English. Cambridge, MA: Harvard University Press.
Bowerman, M. (1973). Structural relationships in children’s utterances: Syntactic orsemantic? In Cognitive development and the acquisition of language. Academic Press.
Braine, M. (1992). What sort of innate structure is needed to bootstrap into syntax?
Cognition, 45, 77–100.
Bresnan, J. (Ed.). (1982). The mental representation of grammatical relations.
Cambridge, MA: MIT Press.
Bresnan, J., & Nikitina, T. (2003). On the gradience of the dative alternation.
Unpublished manuscript. Stanford University.
Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard
University Press.
Brown, R., & Bellugi, U. (1964). Three processes in the child’s acquisition of syntax.
In E. Lenneberg (Ed.), New directions in the study of language (pp. 131–161).
Cambridge, MA: MIT Press.
Buttery, P. (2006). Computational models for first language acquisition (Unpublisheddoctoral dissertation). University of Cambridge.
Calhoun, S. (2010). The centrality of metrical structure in signaling information
structure: A probabilistic perspective. Language, 86, 1–42.
Calhoun, S. (2012). The theme/rheme distinction: Accent type or relative
prominence? Journal of Phonetics, 40, 329–349.
Calhoun, S., Carletta, J., Brenier, J., Mayo, N., Jurafsky, D., & Steedman, M. (2010). The
NXT-format Switchboard corpus: A rich resource for investigating the syntax,
semantics, pragmatics, and prosody of dialog. Language Resources and
Evaluation, 44, 387–419.
Cauvet, E., Limissuri, R., Millotte, S., Skoruppa, K., Cabrol, D., & Christophe, A. (2014).
Function words constrain on-line recognition of verbs and nouns in French 18month-olds. Language Learning and Development, 10(1), 1–18.
Çakıcı, R. (2005). Automatic induction of a CCG grammar for Turkish. In Proceedingsof the student workshop, 43rd annual meeting of the ACL, Ann Arbor, MI(pp. 73–78). ACL.
Chang, N. C.-L. (2008). Constructing grammar: A computational model of theemergence of early constructions. ProQuest.
Charniak, E. (1997). Statistical parsing with a context-free grammar and word
statistics. In Proceedings of the 14th national conference of the American
association for artificial intelligence, Providence, RI., july (pp. 598–603).
Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge, MA: MIT Press.
Chomsky, N. (1981). Lectures on government and binding. Dordrecht: Foris.
Chomsky, N. (1995). The minimalist program. Cambridge, MA: MIT Press.
Christodoulopoulos, C., Goldwater, S., & Steedman, M. (2010). Two decades of
unsupervised POS tagging—How far have we come? In Proceedings of theconference on empirical methods in natural language processing (pp. 575–584).
ACL.
Chrupała, G., Kádár, Á., & Alishahi, A. (2015). Learning language through pictures. In
Proceedings of the 53nd annual meeting of the Association for Computational
Linguistics (pp. 112–118).
Clark, E. (1973). What’s in a word? On the child’s acquisition of semantics in his first
language. In T. Moore (Ed.), Cognitive development and the acquisition of language(pp. 65–110). Academic Press.
Clark, S., & Curran, J. R. (2004). Parsing the WSJ using CCG and log-linear models. In
Proceedings of the 42nd annual meeting of the Association for Computational
Linguistics (pp. 104–111). Barcelona, Spain: ACL.
Cohn, T., Blunsom, P., & Goldwater, S. (2010). Inducing tree-substitution grammars.
The Journal of Machine Learning Research, 11, 3053–3096.
Collins, M. (1997). Three generative lexicalized models for statistical parsing. In
Proceedings of the 35th annual meeting of the Association for Computational
Linguistics (pp. 16–23). Madrid: ACL.
Connor, M., Fisher, C., & Roth, D. (2012). Starting from scratch in semantic rolelabeling: Early indirect supervision. In Cognitive aspects of computational
language acquisition (pp. 257–296). Springer.
Cook, P., & Bildhauer, F. (2011). Annotating information structure: The case of topic.
In Beyond semantics: Corpus based investigations of pragmatic and discourse
phenomena (pp. 45–56). Ruhr Universität, Bochum: Bochumer Linguistische
Arbeitsberichte.
Crain, S., & Nakayama, M. (1987). Structure dependence in grammar formation.
Language, 522–543.
Crocker, M. W., & Keller, F. (2005). Probabilistic grammars as models of gradience inlanguage processing. In Gradience in grammar: Generative perspectives. Oxford,
UK: Oxford University Press.
Croft, W. (2001). Radical construction grammar: Syntactic theory in typologicalperspective. Oxford: Oxford University Press.

142

O. Abend et al. / Cognition 164 (2017) 116–143

Culbertson, J., Smolensky, P., & Wilson, C. (2013). Cognitive biases, linguistic
universals, and constraint-based grammar learning. Topics in Cognitive Science,
5, 392–424.
Dominey, P. F., & Boucher, J.-D. (2005). Learning to talk about events from narrated
video in a construction grammar framework. Artificial Intelligence, 167(1),31–61.
Elman, J., Bates, E., Johnson, M. H., Karmiloff-Smith, A., Parisi, D., & Plunkett, K.
(1996). Rethinking innateness: A connectionist perspective on development.
Cambridge, MA: MIT Press.
Fazly, A., Alishahi, A., & Stevenson, S. (2010). A probabilistic computational model ofcross-situational word learning. Cognitive Science, 34, 1017–1063.
Feldman, N. H., Griffiths, T. L., Goldwater, S., & Morgan, J. L. (2013). A role for thedeveloping lexicon in phone tic category acquisition. Psychological Review, 120(4), 751–778.
Fernald, A. (1993). Approval and disapproval: Infant responsiveness to vocal affectin familiar and unfamiliar languages. Child Development, 64, 657–667.
Fernald, A., Taeschner, T., Dunn, J., Papousek, M., Boysson-Bardies, B., & Fukui, I.
(1989). A cross-language study of pros odic modifications in mothers’ andfathers’ speech to infants. Journal of Child Language, 16, 477–501.
Fisher, C., Gertner, Y., Scott, R., & Yuan, S. (2010). Syntactic bootstrapping. Wiley
Interdisciplinary Reviews: Cognitive Science, 1, 143–149.
Fisher, C., Hall, G., Rakowitz, S., & Gleitman, L. (1994). When it is better to receive
than to give: Syntactic and conceptual constraints on vocabulary growth.
Lingua, 92, 333–375.
Fisher, C., Klingler, S. L., & Song, H.-j. (2006). What does syntax say about space? 2year-olds use sentence structure to learn new prepositions. Cognition, 101(1),B19–B29.
Fisher, C., & Tokura, H. (1996). Prosody in speech to infants: Direct and indirect
acoustic cues to syntactic structure. In J. Morgan & K. Demuth (Eds.), Signal tosyntax: Bootstrapping from speech to grammar in early acquisition (pp. 343–363).
Erlbaum.
Fodor, J. A., Bever, T., & Garrett, M. (1974). The psychology of language. New York:
McGraw-Hill.
Fodor, J. D. (1998). Unambiguous triggers. Linguistic Inquiry, 29, 1–36.
Fodor, J. D., & Sakas, W. (2005). The subset principle in syntax: Costs of compliance.
Journal of Linguistics, 41, 513–569.
Frank, M., Goodman, N., & Tenenbaum, J. (2009). Using speakers’ referential
intentions to model early cross-situational word learning. Psychological Science,
20, 578–585.
Frank, S., Feldman, N., & Goldwater, S. (2014). Weak semantic context helps
phonetic learning in a model of infant language acquisition. In Proceedings of the
52nd annual meeting of the association of computational linguistics.
Furth, H. (1961). The influence of language on the development of concept
formation in deaf children. Journal of Abnormal and Social Psychology, 63,
386–389.
Gazdar, G., Klein, E., Pullum, G. K., & Sag, I. (1985). Generalized phrase structure
grammar. Oxford: Blackwell.
Gentner, D. (1982). Why nouns are learned before verbs: Linguistic relativity versus
natural partitioning. In S. Kuczaj(Ed.). Language development (Vol. 2,
pp. 301–334). Hillsdale, NJ: Erlbaum.
Gentner, D., & Boroditsky, L. (2001). Individuation, relativity, and early wordlearning. In M. Bowerman & S. Levinson (Eds.) (pp. 215–256). Cambridge:
Cambridge University Press.
Gerken, L. (1996). Prosodic structure in young children’s language production.
Langauge, 72, 683–712.
Gerken, L., Jusczyk, P., & Mandel, D. (1994). When prosody fails to cue syntactic
structure. Cognition, 51, 237–265.
Gertner, Y., Fisher, C., & Eisengart, J. (2006). Learning words and rules: abstract
knowledge of word order in early sentence comprehension. Psychological
Science, 17(8), 684–691.
Gibson, E., & Wexler, K. (1994). Triggers. Linguistic Inquiry, 25, 355–407.
Gleitman, L. (1990). The structural sources of verb meanings. Language Acquisition,
1, 1–55.
Gleitman, L., Cassidy, K., Nappa, R., Papafragou, A., & Trueswell, J. C. (2005). Hardwords. Language Learning and Development, 1, 23–64.
Göksun, T., Küntay, A. C., & Naigles, L. R. (2008). Turkish children use
morphosyntactic bootstrapping in interpreting verb meaning. Journal of Child
Language, 35, 291–323.
Goldwater, S., Griffiths, T., & Johnson, M. (2009). A Bayesian framework for word
segmentation: Exploring the effects of context. Cognition, 112, 21–54.
Goldwater, S., & Johnson, M. (2003). Learning OT constraint rankings using amaximum entropy model. In Proceedings of the workshop on variation within
Optimality Theory (pp. 113–122). Stockholm University.
Gómez, R., & Maye, J. (2005). The developmental trajectory of nonadjacentdependency learning. Infancy, 7(2), 183–206.
Griffiths, T., & Tenenbaum, J. (2005). Structure and strength in causal induction.
Cognitive Psychology, 51, 334–384.
Griffiths, T., & Tenenbaum, J. (2006). Optimal predictions in everyday cognition.
Psychological Science, 17, 767–773.
Grimshaw, J. (1981). Form, function and the language acquisition device. In L. Baker& J. McCarthy (Eds.). The logical problem of language acquisition (pp. 165–182).
Cambridge, MA: MIT Press.
Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In
Proceedings of the 2nd meeting of the North American chapter of the Association for
Computational Linguistics (pp. 159–166). Pittsburgh, PA.
Halliday, M. (1967). Intonation and grammar in British English. The Hague: Mouton.

Hayes, B., & Wilson, C. (2008). A maximum entropy model of phonotactics andphonotactic learning. Linguistic Inquiry, 39(3), 379–440.
Hockenmaier, J. (2003). Parsing with generative models of predicate-argument
structure. In Proceedings of the 41st meeting of the Association for Computational
Linguistics, Sapporo (pp. 359–366). San Francisco: Morgan-Kaufmann.
Hockenmaier, J., & Steedman, M. (2002). Generative models for statistical parsing
with Combinatory Categorial Grammar. In Proceedings of the 40th meeting of the
Association for Computational Linguistics (pp. 335–342). Philadelphia.
Hoffman, M., Blei, D., & Bach, F. (2010). Online learning for latent
Dirichlet allocation. Advances in Neural Information Processing Systems, 23,
856–864.
Hyams, N. (1986). Language acquisition and the theory of parameters. Dordrecht:
Reidel.
Johnson, M., & Goldwater, S. (2009). Improving nonparametric Bayesian inference:
Experiments on unsupervised word segmentation with adaptor grammars. In
Proceedings of human language technologies: The 2009 annual conference of thenorth American chapter of the Association for Computational Linguistics.
Jones, B. K. (2015). Learning words and syntactic cues in highly ambiguous contexts(Unpublished doctoral dissertation). University of Edinburgh.
Joshi, A., & Schabes, Y. (1997). Tree-adjoining grammars. In G. Rozenberg & A.
Salomaa (Eds.). Handbook of formal languages (Vol. 3, pp. 69–124). Berlin:
Springer.
Jurafsky, D. (1996). A probabilistic model of lexical and syntactic access anddisambiguation. Cognitive Science, 20(2), 137–194.
Klein, D., & Manning, C. D. (2004). Corpus-based induction of syntactic structure:
Models of dependency and constituency. In Proceedings of the 42nd annual
meeting of the Association for Computational Linguistics (pp. 479–486). Barcelona:
ACL.
Klein, D., & Manning, C. D. (2005). Natural language grammar induction with agenerative constituent-context model. Pattern Recognition, 38, 1407–1419.
Klein, E., & Sag, I. A. (1985). Type-driven translation. Linguistics and Philosophy, 8,
163–2 01.
Krishnamurthy, J., & Mitchell, T. (2014). Joint syntactic and semantic parsing withcombinatory categorial grammar. In Proceedings of the 52nd annual meeting of
the Association for Computational Linguistics (Vol. 1: Long papers, pp. 1188–1198). Baltimore, MD.
Kwiatkowski, T. (2012). Probabilistic grammar induction from sentences andstructured meanings (Unpublished doctoral dissertation). University of
Edinburgh.
Kwiatkowski, T., Goldwater, S., Zettlemoyer, L., & Steedman, M. (2012). A
probabilistic model of syntactic and semantic acquisition from child-directedutterances and their meanings. In Proceedings of the 13th conference of theeuropean chapter of the ACL (EACL 2012) (pp. 234–244). Avignon: ACL.
Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., & Steedman, M. (2010). Inducing
probabilistic CCG grammars from logical form with higher-order unification. In
Proceedings of the conference on empirical methods in natural language processing(pp. 1223–1233). Cambridge, MA: ACL.
Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., & Steedman, M. (2011). Lexical
generalization in CCG grammar induction for semantic parsing. In Proceedings ofthe conference on empirical methods in natural language processing(pp. 1512–1523). Edinburgh: ACL.
Ladd, D. R. (1996). Intonational phonology (2nd edition revised 2008). Cambridge:
Cambridge University Press.
Landau, B., & Gleitman, L. (1985). Language and experience: Evidence from the blindchild. Cambridge, MA: Harvard University Press.
Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106,1126–1177.
Levy, R., Reali, F., & Griffiths, T. L. (2009). Modeling the effects of memory on humanonline sentence processing with particle filters. In Proceedings of the 22nd
conference on neural information processing systems (NIPS).
Lewis, M., & Steedman, M. (2014). A⁄ CCG parsing with a supertag-factored model.
In Proceedings of the conference on empirical methods in natural language
processing (pp. 990–1000). Doha, Qatar: ACL.
MacWhinney, B. (2000). The CHILDES project: Tools for analyzing talk. Mahwah, NJ:
Erlbaum.
Matuszek, C., Fitzgerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012). A joint model oflanguage and perception for grounded attribute learning. In Proceedings of the
29th international conference on machine learning (ICML).
Maurits, L., Perfors, A., & Navarro, D. (2009). Joint acquisition of word order andword reference. In Proceedings of the 31st annual conference of the Cognitive
Science Society (pp. 1728–1733).
McMurray, B., Horst, J. S., & Samuelson, L. K. (2012). Word learning emerges fromthe interaction of online referent selection and slow associative learning.
Psychological Review, 119(4), 831.
Mellish, C. (1989). Some chart-based techniques for parsing ill-formed input. In
Proceedings of the 27th annual meeting of the Association for Computational
Linguistics (pp. 102–109).
Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., & Khudanpur, S. (2010). Recurrent
neural network based language model. In Interspeech 2010, 11th annual
conference of the international speech communication association (pp. 1045–1048).
Mikolov, T., Yih, W.-t., & Zweig, G. (2013). Linguistic regularities in continuous space
word representations. In Proceedings of the 2013 conference of the North
American chapter of the Association for Computational Linguistics: Human
language technologies (pp. 746–751). Atlanta: ACL.
Mintz, T. (2003). Frequent frames as a cue for grammatical categories in childdirected speech. Cognition, 90, 91–117.

O. Abend et al. / Cognition 164 (2017) 116–143
Morris, W. C., Cottrell, G. W., & Elman, J. (2000). A connectionist simulation of theempirical acquisition of grammatical relations. In Hybrid neural systems(pp. 175–193). Springer.
Niyogi, P. (2006). Computational nature of language learning and evolution.
Cambridge, MA: MIT Press.
Niyogi, P., & Berwick, R. (1996). A language learning model for finite parameter
spaces. Cognition, 61, 161–193.
Niyogi, S. (2002). Bayesian learning at the syntax-semantics interface. In Proceedingsof the 24th annual conference of the Cognitive Science Society (Vol. 36, pp. 58–63).
Oléron, P. (1953). Conceptual thinking of the deaf. American Annals of the Deaf, 98,
304–310.
Pearl, L., Goldwater, S., & Steyvers, M. (2010). How ideal are we? Incorporating
human limitations into Bayesian models of word segmentation. In Proceedingsof the 34th annual Boston University conference on child language development.
Somerville, MA: Cascadilla Press.
Perfors, A., Tenenbaum, J., & Regier, T. (2011). The learnability of abstract syntactic
principles. Cognition, 118, 306–338.
Perfors, A., Tenenbaum, J., & Wonnacott, E. (2010). Variability, negative evidence,and the acquisition of verb argument constructions. Journal of Child Language,37, 607–642.
Phillips, L., & Pearl, L. (2014). Bayesian inference as a cross-linguistic word
segmentation strategy: Always learning useful things. In Proceedings of thecomputational and cognitive models of language acquisition and language
processing workshop at EACL.
Pierrehumbert, J., & Hirschberg, J. (1990). The meaning of intonational contours in
the interpretation of discourse. In P. Cohen, J. Morgan, & M. Pollack (Eds.),
Intentions in communication (pp. 271–312). Cambridge, MA: MIT Press.
Pinker, S. (1979). Formal models of language learning. Cognition, 7, 217–283.
Plunkett, K., Sinha, C., Møller, M. F., & Strandsby, O. (1992). Symbol grounding or the
emergence of symbols? Vocabulary growth in children and a connectionist net.
Connection Science, 4(3–4), 293–312.
Pollard, C., & Sag, I. (1994). Head driven phrase structure grammar. Stanford, CA: CSLI
Publications.
Redington, M., Chater, N., & Finch, S. (1998). Distributional information: A powerfulcue for acquiring syntactic categories. Cognitive Science, 22, 425–469.
Regier, T. (2005). The emergence of words: Attentional learning in form andmeaning. Cognitive Science, 29(6), 819–865.
Reznick, J. S., & Goldfield, B. A. (1992). Rapid change in lexical development incomprehension and production. Developmental Psychology, 28(3), 406.
Ross, J. R. (1986). Infinite syntax. Norton, NJ: Ablex.
Saffran, J., Aslin, R., & Newport, E. (1996). Statistical learning by 8-month-old
infants. Science, 274, 1926–1928.
Sagae, K., Davis, E., Lavie, A., MacWhinney, B., & Wintner, S. (2010). Morphosyntactic
annotation of CHILDES transcripts. Journal of Child Language, 37, 705–729.
Sakas, W., & Fodor, J. D. (2001). The structural triggers learner. In S. Bertolo (Ed.),
Language acquisition and learnability (pp. 172–233). Cambridge: Cambridge
University Press.
Sanborn, A. (2017). Types of approximation for probabilistic cognition: Sampling
and variational. Brain and Cognition, 112, 98–101.
Sanborn, A., Griffiths, T., & Navarro, D. (2010). Rational approximations to rationalmodels: Alternative algorithms for category learning. Psychological Review, 117,
1144–1167.
Sato, M.-A. (2001). Online model selection based on the variational Bayes. NeuralComputation, 13(7), 1649–1681.
Schlesinger, I. (1971). Production of utterances and language acquisition. In D.
Slobin (Ed.), The ontogenesis of grammar (pp. 63–101). New York: Academic
Press.
Shi, L., Griffiths, T., Feldman, N., & Sanborn, A. (2010). Exemplar models as amechanism for performing Bayesian inference. Psychonomic Bulletin & Review,
17(4), 443–464.
Siskind, J. (1992). Naive physics, event perception, lexical semantics, and language
acquisition (Unpublished doctoral dissertation). MIT.
Siskind, J. (1996). A computational study of cross-situational techniques forlearning word-to-meaning mappings. Cognition, 61, 39–91.
Snedeker, J., & Gleitman, L. (2004). Why it is hard to label our concepts. In G. Hall &S. Waxman (Eds.), Weaving a lexicon (pp. 257–294). MIT Press.
Sorace, A., & Keller, F. (2005). Gradience in linguistic data. Lingua, 115, 1497–1524.

143

Steedman, M. (1996a). The role of prosody and semantics in the acquisition ofsyntax. In J. Morgan & K. Demuth (Eds.), Signal to syntax (pp. 331–342).
Hillsdale, NJ: Erlbaum.
Steedman, M. (1996b). Surface structure and interpretation. Cambridge, MA: MIT
Press.
Steedman, M. (2000). The syntactic process. Cambridge, MA: MIT Press.
Steedman, M. (2012). Taking scope: The natural semantics of quantifiers. Cambridge,
MA: MIT Press.
Steedman, M. (2014). The surface-compositional semantics of English intonation.
Language, 90, 2–57.
Sundermeyer, M., Schlüter, R., & Ney, H. (2012). LSTM neural networks for language
modeling. In Proceedings of interspeech (pp. 194–197).
Thomforde, E., & Steedman, M. (2011). Semi-supervised CCG lexicon extension. In
Proceedings of the conference on empirical methods in natural language processing(pp. 1246–1256). ACL.
Thompson, C., & Mooney, R. (2003). Acquiring word-meaning mappings for natural
language interfaces. Journal of Artificial Intelligence Research, 18, 1–44.
Thornton, R., & Tesan, G. (2007). Categorical acquisition: Parameter setting in
Universal Grammar. Biolinguistics, 1, 49–98.
Tomasello, M. (1992). First verbs: A case study in early grammatical development.
Cambridge: Cambridge University Press.
Tomasello, M. (1999). The cultural origins of human cognition. Cambridge, MA:
Harvard University Press.
Tomasello, M., & Farrar, M. (1986). Joint attention and early language. Child
Development, 1454–1463.
Trueswell, J., & Gleitman, L. (2007). Learning to parse and its implications forlanguage acquisition. In G. Gaskell (Ed.), Oxford handbook of psycholinguistics(pp. 635–656). Oxford: Oxford University Press.
Turney, P., & Pantel, P. (2010). From frequency to meaning: Vector space models of
semantics. Journal of Artificial Intelligence Research, 37, 141–188.
Ural, A. E., Yuret, D., Ketrez, F. N., Koçbasß, D., & Küntay, A. C. (2009). Morphologicalcues vs. number of nominals in learning verb types in Turkish: The syntactic
bootstrapping mechanism revisited. Language and Cognitive Processes, 24,1393–1405.
Villavicencio, A. (2002). The acquisition of a unification-based generalised categorial
grammar (Unpublished doctoral dissertation). University of Cambridge.
Vygotsky, L. (1934/1986). Thought and language (A. Kozulin, Trans.). Cambridge, MA:
MIT Press.
Wexler, K., & Culicover, P. (1980). Formal principles of language acquisition.
Cambridge, MA: MIT Press.
Xu, F., & Tenenbaum, J. B. (2007). Word learning as Bayesian inference. Psychological
Review, 114(2), 245.
Yang, C. (2002). Knowledge and learning in natural language. Oxford: Oxford
University Press.
Yang, C. (2006). The infinite gift. New York: Scribner.
Yu, C. (2006). Learning syntax–semantics mappings to bootstrap word learning. In
Proceedings of the 28th annual conference of the Cognitive Science Society (pp.
924–929).
Yu, C., & Ballard, D. H. (2007). A unified model of early word learning: Integrating
statistical and social cues. Neurocomputing, 70, 2149–2165.
Yu, C., & Smith, L. B. (2012). Embodied attention and word learning by toddlers.
Cognition, 125(2), 244–262.
Yu, C., & Smith, L. B. (2013). Joint attention without gaze following: Human infants
and their parents coordinate visual attention to objects through eye-handcoordination. PloS One, 8(11), e79659.
Yu, H., & Siskind, J. (2013). Grounded language learning from video described withsentences. In Proceedings of the 51st annual meeting of the Association for
Computational Linguistics (pp. 53–63). Sofia.
Yurovsky, D., Smith, L. B., & Yu, C. (2013). Statistical word learning at scale: Thebaby’s view is better. Developmental Science, 16(6), 959–966.
Zettlemoyer, L., & Collins, M. (2005). Learning to map sentences to logical form:
Structured classification with Probabilistic Categorial Grammars. In Proceedingsof the 21st conference on uncertainty in AI (UAI) (pp. 658–666). Edinburgh: AAAI.
Zettlemoyer, L., & Collins, M. (2007). Online learning of relaxed CCG grammars forparsing to logical form. In Proceedings of the joint conference on empiricalmethods in natural language processing and computational natural language
learning (pp. 678–687). Prague: ACL.