Question: Does the method proposed by the authors outperform neural language models trained with well-tuned hyperparameters?
Evidence: Categories such as animal or furniture are fundamental cognitive building blocks allowing humans to efficiently represent and communicate the complex world around them. Concepts (e.g., dog, table) are grouped into categories based on shared properties pertaining, for example, to their behavior, appearance, or function. Categorization underlies other cognitive functions such as perception BIBREF0 , BIBREF1 or language BIBREF2 , BIBREF3 , and there is evidence that categories are not only shaped by the world they represent, but also by the language through which they are communicated BIBREF4 , BIBREF5 . Although mental categories exist across communities and cultures, their exact manifestations differ BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 . For example, American English speakers prefer taxonomic categorizations (e.g., mouse,squirrel) while Chinese speakers tend to prefer to categorize objects relationally (e.g., tree, squirrel; BIBREF7 ).
Given their prevalent function in human cognition, the acquisition and representation of categories has attracted considerable attention in cognitive science, and numerous theories have emerged BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 . Empirical studies of category acquisition and representation, have been predominantly based on small-scale laboratory experiments. In a typical experiment, human subjects are presented with small sets of often artificial concepts, such as binary strings BIBREF14 or colored shapes BIBREF15 , with strictly controlled features BIBREF16 , BIBREF17 , BIBREF18 . Hypotheses and principles of human categorization are established based on the processes and characteristics of the categorizations produced by the participants. The distribution of subjects participating in such studies is often skewed towards members of cultural and socioeconomic groups which are prevalent in the environment where the research is conducted, and typically consists to a large proportion of western, educated, wealthy and English-speaking participants often sampled from the even more specific population of college students. The demographic and socioeconomic bias has been long recognized, and the question of how this bias might impact conclusions about human cognition in general BIBREF19 and category learning specifically is under active debate BIBREF9 . Although laboratory studies are invaluable for understanding categorization phenomena in a controlled environment, they are also expensive and time-consuming to conduct, and consequently problematic to scale.
In this work, we scale the investigation of category learning and representation along two axes: (1) the complexity of the learning environment, and consequently the richness of learnable concept and category representations, and (2) the diversity of languages and cultures considered in evaluation. We present a novel knowledge-lean, cognitively motivated Bayesian model which learns categories and their structured features jointly from large natural language text corpora in five diverse languages: Arabic, Chinese, English, French, and German. We approximate the learning environment using large corpora of natural language text. Language has been shown to redundantly encode much of the non-linguistic information in the natural environment BIBREF20 , and to influence the emergence of categories BIBREF4 , BIBREF5 . Besides text corpora can cover arbitrarily semantically complex domains, and are available across languages, providing an ideal test environment for studying categorization at scale.
Figure 1 illustrates example input to our model, and Figure 2 shows example categories and associated features as induced by our model from the English Wikipedia. Following prior work BIBREF21 , BIBREF22 , we create language-specific sets of stimuli, each consisting of a mention of target concept (e.g., apple), within its local linguistic context (e.g., {contains, seeds}; cf., Figure 1 ). We consider each stimulus an observation of the concept, i.e., the word referring to the concept is an instance of the concept itself, and its context words are a representation of its features. Our model infers categories as groups of concepts occurring with similar features; and it infers feature types as groups of features which co-occur with each other. The output of our model (cf., Figure 2 ) are categories as clusters of concepts, each associated with a set of feature types, i.e., thematically coherent groups of features. We train a separate model on each of our target languages, each time presenting the model with input stimuli from the relevant language.
Computational models in general, and Bayesian models in particular, allow to investigate hypotheses about cognitive phenomena by systematically modifying the learning mechanism or available input while observing the learning outcome. Bayesian models have been applied to a variety of cognitive phenomena BIBREF23 , BIBREF24 , BIBREF25 , and category acquisition is no exception. Following from Anderson's BIBREF14 , BIBREF26 , BIBREF27 seminal work, a number of models have been developed, and tested in their ability to reproduce human behavior in laboratory settings by exposing the models to small sets of controlled inputs with restricted features. In this work we draw on the full potential of computational modeling by exposing our models to (a) more complex data reflecting the diversity of contexts in which concepts can be observed; and (b) a input data in different languages, shedding light on the applicability of computational cognitive models beyond the prevalent English test language.
Categorization tasks in a laboratory environment typically involve stimuli with a small set of features which are relevant to the categorization target, eliminating the need to detect features, and discriminate them in their relevance. In the real world, however, concepts are observed in contexts and a substantial part of acquiring categorical knowledge involves learning which features are useful to discriminate among concepts. In fact, research has shown that humans learn features jointly with categories BIBREF28 , BIBREF29 and that these features are themselves structured so as to represent the diversity and complexity of the properties exhibited in the world BIBREF30 , BIBREF31 , BIBREF32 . Our novel model of category learning presented in this article, jointly learns categories and their structured features from large sets of informationally rich data.
Our work exemplifies the opportunities that arise from computational models and large data sets for investigating the mechanisms with which conceptual representations emerge, as well as the representations themselves in a broader context. We simulate the acquisition of categories comprising hundreds of concepts by approximating the learning environment with natural language text. Language has been shown to redundantly encode much of the non-linguistic information in the natural environment BIBREF20 , as well as human-like biases BIBREF33 , and to influence the emergence of categories BIBREF4 , BIBREF5 . Text corpora are a prime example of naturally occurring large-scale data sets BIBREF34 , BIBREF35 , BIBREF36 . In analogy to real-world situations, they encapsulate rich, diverse, and potentially noisy, information. The wide availability of corpora allows us to train and evaluate cognitive models on data from diverse languages and cultures. We test our model on corpora from five languages, derived from the online encyclopedia Wikipedia in Arabic, Chinese, French, English, and German. Wikipedia is a valuable resource for our study because it (a) discusses concepts and their properties explicitly and can thus serve as a proxy for the environment speakers of a language are exposed to; and (b) allows us to construct corpora which are highly comparable in their content across languages, controlling for effects of genre or style.
We present a series of evaluations investigating the quality of the induced categories and features. Leveraging a reference comprising hundreds of concepts and more than 30 categories, we demonstrate that our model learns meaningful categories in all five target languages. We furthermore show, through crowd-sourced evaluations involving native speakers of each target language, that the induced feature types are (a) each thematically coherent and interpretable; and (b) are associated with categories in comprehensible ways. We discuss language-specific idiosyncrasies emerging from the induced representations.
In the remainder of this article, we first review related literature, before we present a cognitively motivated model for learning categories and their structured representations from large natural language corpora. We then evaluate the quality of the emerging representations, as well as the generalizability of our model across languages. Note that the primary goal of this work is not to characterize differences in categories and features arising from different languages (even though this would be an interesting avenue for future work). Rather, we aim to demonstrate the utility of large-scale naturalistic datasets for cognitive modeling, and to verify mechanisms of categorization known from laboratory studies at scale and across communities.
Our experiments indicate that the proposed method outperforms neural language models trained with well-tuned hyperparameters and achieves state-of-the-art scores on each dataset.
Answer: Yes, it does.