Learning Categories Using Semantic Priming in a Bayesian Framework

A probabilistic algorithm for unsupervised learning of categories using internal comparisons is presented. The Bayesian merging algorithm uses parent category prototypes to facilitate basic category comparisons. Our model takes on the new task of learning internal representation with no explicit feedback. We assume that humans make implicit comparisons between novel categories and merge categories they consider similar, in a probabilistic sense. We find that the algorithm out performs simple data incorporation, and that the algorithm supplemented by feature comparisons did even better. The consideration of context brought about by feature comparison allows the model to generalize over noisy examples. However, a more compact representation is desired. Introduction Humans can learn nouns easily given a few examples and almost no explicit feedback. A simple explicit memory model can account for neither the rapidity nor the accuracy of learning. Thus we must rely on a model of implicit memory that accounts for the effects of past experience. One approach is to represent nouns as basic categories with uncertain feature values. As more data are incorporated, the categories look more and more like what the noun represents in the real world. Since categories are built up from experience, we expect physical interaction to improve category acquisition. We suggest that implicit category comparisons (via semantic priming) model this physical interaction. Much of the modeling ideas presented are inspired by Schooler et. al. (2001). As in Schooler et. al., we apply rational analysis (Anderson, 1990) to the modeling of results from psychological experiments. The REM model of Schooler et. al. was an attempt to account for episodic retrieval via a model that assumes that priming acts to “alter the word’s lexical-semantic memory representation” (2001). Justification for direct alteration of the representation are given in Schooler et. al. Our priming effects will be based on comparison tasks. Most objections Submitted to NIPS Neural Information Process Systems to Bayesian models point to the lack of plausibility for the human mind to perform probabilistic computations during experimental tasks. Our model will make simplifying assumptions on the REM model by using simpler Bayesian methods that we assume are at least can be computed neurally. Moreover, we will apply Bayesian reasoning to learning a best current model. We will simplify the algorithm by making some assumptions on human perceptual capabilities and mental representations. We focused on animal categories so as to simplify direct comparison with psychological experiments. Ideas suggested here are relevant for text categorization as well. In particular, any domain that can be decomposed into hierarchical categories can be modeled. Our algorithm uses a simple representation of features and categories to facilitate comparison. In a real-world application, Bayesian network representations should be constructed for both basic and parent categories. We’ll suggest how this can be done later, although the implementation is only at an experimental phase. Assumptions First, we assume that no perceptual errors are made during learning and testing. That is, there is no probability distribution associated with input animal features for each example. The lack of errors in perception does not, however, imply that the input data is noiseless. We assume that the distributions of attributes of an animal are independent and jointly Gaussian. Data are generated using normal distributions with pre-determined means and variances that reflect the attribute domain for each animal. Noisy examples with random features are also fed into the model periodically. The assumption suggests that the only probabilistic aspect of category-learning is captured by our model. Thus, our model (and not the perceptual system) is responsible for handling noisy data. Second, only two levels of categories are relevant to our task. Our basic categories consist of animal types. In particular, data for ant, snail, frog, cat, wolf, cow, and whale were given as input. Note, however, that the model forms abstract models of the data, so no actual label is given to any category. The idea is that a category is described by average feature values and a measure of the spread of feature values. It is the model’s task to minimize the complexity of sets of categories while maximizing the probability of correctly identifying the category of any example given the feature values. This can be done using inference on a belief network. Our superordinate parent categories consist of graded conceptual categories involving the animal types. For example, the large animal category should include whale and cow as salient examples, with wolf, and possibly cat having graded membership. Graded membership is not represented explicitly, however. The model will keep track of one best prototype, which can be updated with a given probability during comparison tasks. We construct conceptual categories using the different features associated with animals. In particular, large animal, small animal, cute animal, hideous animal, ferocious animal, and tame animal were used. Ideally, we should model other objects as well and allow our conceptual categories to be simply large, small, cute, hideous, etc. This makes the model more general, less domain-dependent, but harder to assess. Thus we stick to animals at present. Third, inferences and comparisons done on particular members of basic categories are necessary for learning and structuring both basic and superordinate parent categories. Here we focus on feature value comparisons. We can ask, for example, “which animal is larger, cat or frog? ” Luo (2000) reviews these tasks. In modeling early development, we can associate the semantic priming that results from querying the subject with changes in the underlying categories. Our claim is that these changes result from comparison of basic category feature values with prototype feature values. In the example above, size feature values for cat and frog are compared with the prototype large animal feature value. Changes are made to the abstract categorical representations of cat and frog size values and spreads. Prototype values are updated as needed. One natural question that arises is: if feature value estimates are already kept by our basic category representation, couldn’t we just compare the estimates and return the appropriate category (in this case, cat for the larger animal)? There are three problems with this approach. First, the approach predicts that all comparisons should take the same amount of time on average. This has been shown to be false (Banks et. al., 1976). For example, it is much faster to decide between whale and ant than cat and frog. Second, the approach predicts that no errors could occur. For example, frogs would always be judged smaller than cats. This does not hold experimentally (Luo, 2000). We need, instead, a model that is generally true, but can give wrong answers in some contexts. Third, there are uncertainties associated with feature estimates (This is why we need a measure of spread). For example, some cats (tigers) are very large while other cats (kitty) are very small. As more examples of cats are shown to the system, the model becomes better and better at estimating the relative size of cats, but it is never absolutely sure that cats are smaller than wolves. Our approach is to compare cat and wolf sizes with the prototypical size. We can model differences in judgment reaction times by relative distances between category and prototype size feature values. You can think of our approach as a variance reduction technique. We are leveraging our knowledge about prototypical cases, which also have associated uncertainty values. Fourth, there exist abstract category representations in the mind independent of, but influenced by, linguistic labels. Our model keeps abstract categories with no particular linguistic interpretation. The reason for this is that synonyms abound in the real world. While there is evidence that no “exact” synonyms exist in the technical sense, our model of the mind relies on generalization over similar word senses. Hence, frog and toad may refer to relatively similar animals that have similar values with respect to our set of features. A child may not be able to distinguish (at least at first) between these similar animal types. Thus we need a structure that represents the features of a frog-toad without labeling it. We can then compare these abstract categories amongst themselves and perform inferences purely on the basis of feature structures. Words with different senses (cat, for example) are described by different categories while different words denoting animals with similar features (frog and toad, for example) are described by the same category. We hope to show in the future that the linguistic label associated with an abstract category is a probability distribution over words or phrases acquired from experience. Instead of saying, for example, that the word cat has two senses (kitty and tiger), we say that abstract category A has a probability of 0.95 of being labeled “cat” and abstract category B has a probability of 0.5 of being labeled “cat. ” With experience, the abstract categories become finer because the variances of the feature values become smaller. In the limit of infinite experience, the two word senses of “cat” will be described by separate abstract categories with labeling probability 1.0, because a separate abstract category will describe “tiger” with probability 1.0. We hope that someone will prove the following claim (it shouldn’t be hard): Given a pre-defined, finite, sufficiently “nice” set of training examples, the category partitions formed by the algorithm will converge in probability to a set of non-overlapping category distributions (in the sense of zero variance feature values