Testing the psychological validity of cluster construction biases

Testing the psychological validity of cluster construction biases Joseph L. Austerweil (Joseph Austerweil@Brown.edu) Brown University, Department of Cognitive, Linguistic, and Psychological Sciences, Providence, RI 02912 Abstract not due to lack of interest, but rather the difficulty of empir- ically comparing these models. Some challenges include the combinatorial explosion in the number of possible clusterings for even a small number of items (e.g., there are more than 100,000 possible clusterings for 10 items; Pitman, 2002), the indirect and at times weak effect of cluster construction pro- cesses, and the sensitivity of the clusters reported by partici- pants to instructions (e.g., providing a number of clusters af- fects participants’ responses; Pothos et al., 2011). In this article, I present one of the first empirical experi- ments that directly tests the psychological validity of com- putational models that construct clusters to group observa- tions. This new approach avoids the combinatorial explo- sion of having to test an insurmountably large number of stimuli on some measure. Instead, I test people’s prior ex- pectations for constructing clusters and compare them to model predictions using an adaptive experimental technique, Markov chain Monte Carlo with People (MCMCP, which is also called iterated learning; Griffiths & Kalish, 2007; San- born, Griffiths, & Shiffrin, 2010). In this method, participant responses are used to guide the experiment towards testing clusters that are consistent with their prior expectations. Pre- vious work has shown that MCMCP is especially effective for eliciting peoples prior expectations over a large set of possi- bilities despite the combinatorial explosion (e.g., happy and sad faces; Martin, Griffiths, & Sanborn, 2012). In this article I first describe previous work testing models that construct clusters. Next, I explain MCMCP and present the experimental methods. Then, I describe the models tested in this paper (Bayesian nonparametric models, SUSTAIN, and a few alternatives). Last, I discuss the results and con- clude with implications and directions for future work. To generalize from one experience to the next in a world where the underlying structures are ever-changing, people construct clusters that group their observations and enable information to be pooled within a cluster in an efficient and effective man- ner. Despite substantial computational work describing poten- tial domain-general processes for how people construct these clusters, there has been little empirical progress comparing dif- ferent proposals to each other and to human performance. In this article, I empirically test some popular computational pro- posals against each other and against human behavior using the Markov chain Monte Carlo with People methodology. The re- sults support two popular Bayesian nonparametric processes, the Chinese Restaurant Process and the related Dirichlet Pro- cess Mixture Model. Keywords: Clustering, Bayesian modeling, Connectionist modeling, Categorization Introduction A child observing a 3-tined fork for the first time easily infers its function. She infers that it is more similar to the 4-tined forks she typically uses than to other eating utensils such as spoons and knives. To generalize information from one ob- servation (e.g., a 4-tined fork) to another observation (e.g., a 3-tined fork), people group their observations into clusters (e.g. forks). A cluster summarizes the information common to the stimuli within it, which enables efficient generaliza- tion and learning. In the example above, the child represents the category of utensils as a combination of three clusters: forks, spoons, and knives. A novel property learned about one fork should be generalized more to other forks (within- cluster) than to other utensils that are not forks. Given that underlying structures of the world are ever-changing and in- dividuals only observe limited information, the clusters that encode these structures need to be flexible to accommodate new types of items. However, a new item can be too different to fit in any existing cluster, and a new cluster must be con- structed. For example, when the child sees her first pair of chopsticks, she knows they are part of the category utensils, but they do not fit within her current set of clusters (forks, spoons, and knives). So she must create a new cluster to rep- resent chopsticks. How does the mind construct clusters to represent observations? Cognitive scientists have proposed several domain-general cluster construction processes with in a variety of domains, such as categorization (Anderson, 1991; Love, Medin, & Gureckis, 2004), associative learning (Gershman, Blei, & Niv, 2010), causal inference (Kemp, Tenenbaum, Niyogi, & Griffiths, 2010), perceived orientation (Austerweil, Friesen, & Griffiths, 2011), and word segmentation (Goldwater, Grif- fiths, & Johnson, 2009). Despite the proliferation of differ- ent proposals, there have been few attempts to compare the proposals and evaluate their psychological validity. This is Previous Empirical Work There are few empirical tests comparing human behavior to different computational models that construct clusters. One exception is Pothos et al. (2011), who tested how people construct clusters in the domain of unsupervised categoriza- tion without instructing participants to use a certain num- ber of clusters. Participants were asked to sort nine two- dimensional stimuli (spider-like images varying in the length of their “body” and “legs”) into groups that felt intuitive. The authors compared participant sortings to their “goodness” ac- cording to several unsupervised category learning models. Their results were equivocal, but generally, they supported SUSTAIN and a few other models better than the Rational Model of Categorization (a Dirichlet process mixture model using a particular approximation method; Anderson, 1991; Neal, 1998). Their results are an informative first comparison of human and model clustering in unsupervised categoriza-