People ignore token frequency when deciding how widely to generalize

People ignore token frequency when deciding how widely to generalize Amy Perfors (amy.perfors@adelaide.edu.au) Keith Ransom (keith.ransom@adelaide.edu.au) Daniel J. Navarro (daniel.navarro@adelaide.edu.au) School of Psychology, University of Adelaide Abstract Many theoretical accounts of generalization suggest that with increasing data, people should tighten their generalizations. However, these accounts presume that the additional data points are all distinct. Other accounts, such as the adap- tor grammar framework in linguistics (Johnson, Griffiths, & Goldwater, 2007), suggest that when the additional data points are identical, generalizations about grammaticality need not tighten appreciably: they may be made on the basis of type fre- quency rather than token frequency (although token frequency can affect other types of learning). We investigated what hap- pens in this situation by presenting participants with identi- cal data in both a linguistic and a non-linguistic context, some ten times as much as others, and asking them to generalize to novel exemplars. We find that people are insensitive to token frequencies when determining how far to generalize, though memory has a small mediating effect: generalizations tighten slightly more when people may rely on a memory aid. Keywords: generalization; category learning; adaptor gram- mar; grammar learning; types; tokens; size principle; fre- quency Introduction How far should people generalize from the data they have ob- served? If a child points out two beagles and a basset hound and calls them GLUGGIES , should we guess that she will also call a great dane by this name? We could generalize narrowly, and guess that the word GLUGGIES refers only to small dogs. Or we could generalize broadly, and guess that it might in- clude cats and other typical household pets. One approach to this problem is to assume that observa- tions are sampled randomly from the true extension of the cat- egory, and use Bayes’ rule to guide inferences (Tenenbaum & Griffiths, 2001). According to this “strong sampling” scheme, if a category contains k items, each item will be observed with probability 1 k . Given two hypotheses about the category that are both consistent with the observations, one narrow and the other broad, a Bayesian learner will eventually learn to prefer the narrower hypothesis as more and more data arrive. This is because the narrower hypothesis assigns higher probability to the observations. This is known as the size principle, and it (or something like it) has been shown to guide human gen- eralization in a variety of contexts (Xu & Tenenbaum, 2007; Navarro & Perfors, 2010; Navarro, Dry, & Lee, 2012). One ambiguity in this research, however, is that it assumes that all observations are independently sampled and meaning- fully distinct from one another. Suppose it turned out that the “two” beagles labelled GLUGGY by the child look identical and may be the same animal. Should this be treated as one data point or two? This is a common problem: people do re- peatedly encounter the same dog or hear the same sentence many times (e.g., “How are you? ”). The normative treatment of identical data points depends on how one interprets the generative process behind the data. If the sampling process is something akin to drawing examples with replacement from a bag of possibilities, then the size principle should apply: seeing the same example multiple times is actually strong ev- idence that the true extension of the category is very small. A more conservative approach is for people to make a dis- tinction between distinct entities (types) and the set of in- stances (tokens) on which they have been observed. It is not unreasonable to assume that distinct types represent distinct samples from the category, but multiple tokens of the same type may not. The issue comes down to the relevance of token frequency to the inference at hand. For instance, seeing the same beagle repeatedly may be informative about the number of dogs in the neighborhood, but it does not say much about how common beagles are relative to other kinds of dog. In linguistics this insight is captured using the adaptor grammar framework (Johnson et al., 2007), in which the learner makes a distinction between the underlying category to be learned (the grammar) and a mechanism that shapes the frequencies with which category members are observed (the adaptor). This framework has been successfully applied to many aspects of language (e.g., Johnson & Demuth, 2010; O’Donnell, Snedeker, Tenenbaum, & Goodman, 2011; Per- fors, Tenenbaum, & Regier, 2011), and under certain param- eter values it makes a different prediction than the one that emerges from strong sampling. If the same set of entities are observed many times, this is not in and of itself evidence that the true category is small: generalizations should only tighten when new types are observed, not new tokens. Although the adaptor grammar approach was developed to explain linguistic phenomena, there is no reason why it should not apply more broadly. Categories can be considered to consist of a set of entities, or types (the extension of the category), and a frequency distribution can then be defined over that extension; the former is analogous to the grammar, the latter to the adaptor. On the other hand, language is dif- ferent from concept learning in many ways – for instance, individual sentences are not physical entities in the same way that exemplars from a concept are. It is also possible that token frequencies are relevant in category learning problems but not language learning ones. This paper investigates how people change their general- izations when they encounter new tokens of old types. We begin by presenting an experiment in which participants were shown a dataset of ten distinct types of exemplars, each oc- curring either once or ten times. Our main question is whether people tighten their generalizations when they are shown ten