Interpretation of Generalization in Masked Language Models: An Investigation Straddling Quantifiers and Generics
Authors: Claudia Collacciani, Giulia Rambelli (Università di Bologna, 2023)
Topic: Lexical and Semantic Resources and Analysis

Abstract
Generics are statements that express generalizations and are used to communicate generalizable knowledge. While generics convey general truths (e.g., "Birds can fly"), they often allow for exceptions (e.g., "penguins do not fly"). Nonetheless, generics form the basis of how we communicate our commonsense about the world. We explored the interpretation of generics in Masked Language Models (MLMs), building on psycholinguistic experimental designs. As this interpretation requires a comparison with overtly quantified sentences, we investigated: i) the probability of quantifiers, ii) the internal representation of nouns in generic vs. quantified sentences, and iii) whether the presence of a generic sentence as context influences quantifiers' probabilities. The outcomes confirm that MLMs are insensitive to quantification; nevertheless, they appear to encode a meaning associated with the generic form, which leads them to reshape the probability associated with various quantifiers when the generic sentence is provided as context.

Introduction
Generic generalizations, or generics, are sentences such as "Birds fly" and "Cars have four wheels," which allow us to convey information about categories, or kinds, of individuals. They are used to communicate information that extends beyond the present context and express our knowledge about the world, including beliefs, stereotypes, or prejudices (e.g., "Women are more sensitive than men," as well as the less harmful "Italians eat spaghetti"). Generics can be considered one of the cornerstones of human cognition since they allow us to conceptualize the properties we attribute to categories and thus organize our experience of the world. The most distinctive feature of generics is that they allow for exceptions. For example, "Birds fly" is judged true even if there are birds that cannot fly (e.g., penguins); in this case, therefore, the corresponding universal statement ("All birds fly") is false. Different generalizations tolerate exceptions to varying degrees. Thus, some generic statements might be better paraphrased with "all," others with "most," and others with "some," but—unlike quantified statements—they do not explicitly contain information about the prevalence of the property in the category (i.e., how many members of the category possess the property). Similarly, there is no unambiguous relationship between the prevalence of a property among category members and the acceptability of the corresponding generic as true. For example, the generalization "Lions have manes" is accepted even if only male adult lions have manes, but the generalization "Lions are males" is rejected. Given these properties, the meaning of generalizations can be considered "vague," and their interpretation can be assumed to be derived by people through world knowledge and pragmatic skills. Most of the experimental studies conducted on generics are cognitively driven and based mainly on contrasting generics with overtly quantified sentences; in other words, quantifiers are used to approximate the vague meaning of generics. In this paper, we investigate the interpretation of generalizations in Large Language Models (LLMs) of the Transformer family, building on psycholinguistic experimental designs (in particular, Leslie et al. and Cimpian et al.). Since comparison with quantification seems to be necessary to decode the meaning of generics, we also used quantifiers. We present three tasks related to different but complementary research questions: 1) Are LLMs biased towards some quantifiers more than others? We computed the probability of several quantifiers appended to a generic statement. This analysis serves as a baseline to understand the probability distribution of quantifiers and if there is any bias towards some of them (for example, a generic overgeneralization effect such as that found in humans by Leslie et al.). 2) Are the hidden representations of generics similar to those of quantified phrases? We extracted the hidden representation of words in generic and quantified sentences and compared their representation pairwise to understand which quantified nominal phrase approximates the meaning of generics better. 3) Are LLMs showing the same prevalence effect as humans? We reproduced the experimental design of Cimpian et al.'s Implied Prevalence task to test whether the presence of the generic as a premise impacts the probability of selected quantifiers. The data and code that we used for the experiments are publicly available.

Conclusion
In this paper, we analyzed the interpretation of generics in Masked Language Models (MLMs) through psycholinguistic experimental designs that exploited quantified expressions to investigate the understanding of generic ones. The first two experiments raise questions about the codification of quantifiers, as it seems that the models do not substantially exhibit a strong sensitivity to quantifiers and do not encode a semantic difference in the representation of quantification. Altogether, our results suggest that the models do not appear to contain the commonsense knowledge required to interpret generics that differ in content through quantifiers. However, they seem to have encoded a meaning associated with the generic form, which leads them to reshape the probability associated with various quantifiers when the generic sentence is provided as context. In the last experiment, we observed that the models prefer the universal quantifier unanimously if preceded by a generic utterance. People behave similarly when tested on novel categories, that is, non-word categories for which subjects have no prior understanding. However, people can modulate their interpretations of generalizations in a real language setting through their world knowledge of real categories. Regardless, MLMs tend to treat real and invented categories equally, being agnostic to world knowledge. For this reason, this could be a potentially harmful bias. The presented analysis has theoretical and methodological implications. First, we observed that the language of generalization is a complex phenomenon that is hard to investigate in human processing and even more in LLMs, mostly because the investigation of generics' interpretation makes use of quantifiers, and language models often fail in tasks related to quantification. Another problem lies in the fact that it is difficult to test autoregressive models (e.g., GPT family) on tasks such as the one used in Experiment 1 because, as they do not have access to the right context, they do not have sufficient information to modulate the probabilities associated with the various quantifiers accordingly. Finding ways to test autoregressive models in addition to MLMs would be desirable. In this paper, we have not directly investigated the aspect of the attention in MLMs. Future research could address this aspect. Furthermore, future work could involve the definition of alternative tasks for investigating generalizations to make comparing models and human interpretations easier. Psycholinguistic tests on this phenomenon often rely on truth judgments, and we should be cautious about comparing human truth judgments with model outputs since they lack commonsense knowledge comparable to that of humans. Overall, further investigations are needed to clarify the interpretation of generics in language models.