How Much Competence Is There in Performance? Assessing the Distributional Hypothesis in Word Bigrams
Authors: Johann Seltmann, Luca Ducceschi, Aurélie Herbelot (University of Potsdam, Università di Trento, 2019)
Topic: Distributional Semantics

Abstract
The field of Distributional Semantics (DS) is built on the ‘distributional hypothesis’, which states that meaning can be recovered from statistical information in observable language. It is however notable that the computations necessary to obtain ‘good’ DS representations are often very involved, implying that if meaning is derivable from linguistic data, it is not directly encoded in it. This prompts questions related to fundamental questions about language acquisition: if we regard text data as linguistic performance, what kind of ‘innate’ mechanisms must operate over that data to reach competence? In other words, how much of semantic acquisition is truly data-driven, and what must be hard-encoded in a system’s architecture? In this paper, we introduce a new methodology to pull those questions apart. We use state-of-the-art computational models to investigate the amount and nature of transformations required to perform particular semantic tasks. We apply that methodology to one of the simplest structures in language: the word bigram, giving insights into the specific contribution of that linguistic component.

Introduction
The traditional notions of performance and competence come from Chomsky’s work on syntax (Chomsky, 1965), where much emphasis is put on the mental processes underpinning language acquisition. Chomsky posits the existence of a Universal Grammar, innate in the human species, which gets specialised to the particular language of a speaker. By exposure to the imperfect utterances of their community (referred to as performance data), an individual configures their UG to reach some ideal knowledge of that community’s language, thereby reaching competence. The present paper borrows the notions of ‘performance’, ‘competence’ and ‘innateness’ to critically analyse the semantic ‘acquisition’ processes simulated by Distributional Semantics models (DSMs). Our goal is to tease apart how much of their observed competence is due to the performance data they are exposed to, and how much is contributed by ‘innate’ properties of those systems, i.e. by their specific architectures. DSMs come in many shapes. Traditional unsupervised architectures rely on counting co-occurrences of words with other words or documents (Turney and Pantel, 2010; Erk, 2012; Clark, 2012). Their neural counterparts, usually referred to as ‘predictive models’ (Baroni et al., 2014) learn from a language modelling task over raw linguistic data (e.g. Word2Vec, Mikolov et al., 2013, GloVE Pennington et al., 2014). The most recent language embedding models (Vaswani et al., 2017; Radford et al., 2018), ELMo (Peters et al., 2018), or BERT (Devlin et al., 2018) compute contextualised word representations and sentence representations, yielding state-of-the-art results on sentence-related tasks, including translation. In spite of their differences, all models claim to rely on the Distributional Hypothesis (Harris, 1954; Firth, 1957), that is, the idea that distributional patterns of occurrences in language correlate with specific aspects of meaning. The Distributional Hypothesis, as stated in the DSM literature, makes semantic acquisition sound like an extremely data-driven procedure. But we should ask to what extent meaning indeed is to be found in statistical patterns. The question is motivated by the observation that the success of the latest DSMs relies on complex mechanisms being applied to the underlying linguistic data or the task at hand (e.g. attention, self-attention, negative sampling, particular objective functions). Such mechanisms have been shown to apply very significant transformations to the original input data: for instance, the Word2Vec objective function introduces parallelisms in the space that make it perform particularly well on analogy tasks (Gittens et al., 2017). Models such as BERT apply extensive processing to the input through stacks of encoders. So while meaning can be derived from training regimes involving raw data, it is not directly encoded in it. Interestingly, Harris himself (Harris, 1954) points out that a) distributional structure is in no simple relation to the structure of meaning; b) different distributions in language encode different phenomena with various levels of complexity. We take both points as highlighting the complex relation between linguistic structure and the cognitive mechanisms that are necessary to apply to the raw input to retrieve semantic information. The point of our paper is to understand better what is encoded in observable linguistic structures (at the level of raw performance data), and how much distortion of the input needs to be done to acquire meaning (i.e. what cognitive mechanisms are involved in learning semantic competence). In the spirit of Harris, we think it is worth investigating the behaviour of specific components of language and understand which aspects of meaning they encode, and to what extent. The present work illustrates our claim by presenting an exploratory analysis of one of the simplest recoverable structure in corpora: the word bigram. Our methodology is simple: we test the raw distributional behaviour of the constituent over different tasks, comparing it to a state-of-the-art model. We posit that each task embodies a specific aspect of competence. By inspecting the difference in performance between the simplest and more complex models, we get some insight into the way a particular structure (here, the bigram) contributes to the acquisition of specific linguistic faculties. The failures of raw linguistic data to encode a particular competence points at some necessary, ‘innate’ constraint of the acquisition process, which might be encoded in a model’s architecture as well as the specific task that it is required to solve. In what follows, we propose to investigate the behaviour of the bigram with respect to three different levels of semantic competence, corresponding to specific tasks from the DS literature: a) word relatedness; b) sentence relatedness; c) sentence autoencoding (Turney, 2014; Bowman et al., 2016). The first two tasks test to which extent the linguistic structure under consideration encodes topicality: if it does, it should prove able to cluster together similar lexical items, both in isolation and as the constituents of sentences. The third task evaluates the ability of a system to build a sentence representation and from that representation alone, recover the original utterance. That is, it tests distinguishability of representations. Importantly, distinguishability is at odds with the relatedness tasks which favour clusterability. The type of space learned from the raw data will necessarily favour one or the other. Our choice of tasks thus allows us to understand which type of space can be learned from the bigram: we will expand on this in our discussion.

Conclusion
So what can be said about bigrams as distributional structure? They encode a very high level of lexical discrimination while accounting for some basic semantic similarity. They of course also encode minimal sequential information which can be used to retrieve local sentence ordering. Essentially, they result in representations that are perhaps more ‘symbolic’ than continuous. It is important to note that the reasonable correlations obtained on relatedness tasks were achieved after application of PMI weighting, implying that the raw structure requires some minimal preprocessing to generate lexical information. On the back of our results, we can draw a few conclusions with respect to the relation of performance and competence at the level of bigrams. Performance data alone produces very distinct word representations without any further processing. Some traces of lexical semantics are present, but require some hard-encoded preprocessing step in the shape of the PMI function. We conclude from this that as a constituent involved in acquisition, the bigram is mostly a marker of the uniqueness of word meaning. Interestingly, we note that the notion of contrast (words that differ in form differ in meaning) is an early feature of children’s language acquisition (Clark, 1988). The fact that it is encoded in one of the most simple structures in language is perhaps no coincidence. In future work, we plan a more encompassing study of other linguistic components. Crucially, we will also investigate which aspects of state-of-the-art models such as W2V contribute to score improvement on lexical aspects of semantics. We hope to thus gain insights into the specific cognitive processes required to bridge the gap between raw distributional structure as it is found in corpora, and actual speaker competence.