SlowGuess/NAACL_2022_OCR huggingface dataset:
Mikael Brunila
mikael.brunila@gmail.com
Jack LaViolette
jack.laviolette@columbia.edu
The power of word embeddings is attributed to the linguistic theory that similar words will appear in similar contexts. This idea is specifically invoked by noting that "you shall know a word by the company it keeps," a quote from British linguist J.R. Firth who, along with his American colleague Zellig Harris, is often credited with the invention of "distributional semantics." While both Firth and Harris are cited in all major NLP textbooks and many foundational papers, the content and differences between their theories is seldom discussed. Engaging in a close reading of their work, we discover two distinct and in many ways divergent theories of meaning. One focuses exclusively on the internal workings of linguistic forms, while the other invites us to consider words in new company—not just with other linguistic elements, but also in a broader cultural and situational context. Contrasting these theories from the perspective of current debates in NLP, we discover in Firth a figure who could guide the field towards a more culturally grounded notion of semantics. We consider how an expanded notion of "context" might be modeled in practice through two different strategies: comparative stratification and syntagmatic extension.
We are in the world and the world is in us.
(1938; cited in Firth 1957c, 29)
If you have read any papers in computational linguistics in the past thirty years, you have likely come upon the following quote from British linguist J.R.Firth (1957c, 11): "You shall know a word by the company it keeps". Cited in most major textbooks (Manning and Schütze, 1999; Jurafsky and Martin, 2009; Eisenstein, 2019; Russell and Norvig, 2020), several foundational papers, and hundreds of other NLP articles, this phrase has come to index a theoretical orientation in a
field that is increasingly focused on computation, often at the expense of linguistic theory (on these trends in NLP see Halevy et al., 2009; Manning, 2015; Norvig, 2012; Henderson, 2020; Church and Liberman, 2021). Together with American linguist Zellig Harris, Firth is regularly called upon to justify a distributional theory of semantics, whereby the meaning of lexical units is conceived in terms of relative co-occurrence and shared contexts of use (Sahlgren, 2008).
While Harris and Firth are often invoked, their ideas are seldom closely engaged. Hailing from disparate traditions, Harris and Firth had radically different ideas on the scope and context of linguistic analysis, and presented incongruent versions of the distributional method. Drawing on the information theory pioneered by Claude Shannon (1948), Harris was determined to work out a structuralist theory of language in terms of mathematical information (Léon, 2011; Nevin, 1993). Firth, on the other hand, came to linguistics via anthropology and borrowed heavily from pragmatic philosophies of language. For him, linguistic analysis always started with the "context of situation" and necessarily accounted for non-verbal actors and objects (Firth, 1957c, 9).
Considering the definition and extent of linguistic context is important for many reasons, as a spate of recent publications suggests (Glenberg and Robertson, 2000; Hovy, 2018; Bender and Koller, 2020; Bisk et al., 2020; Tamari et al., 2020; Trotter et al., 2020). Firstly, it touches upon the limits of current paradigms in NLP, where corpus linguistics is perfected through increasingly complex models trained on increasingly massive corpora (Bender et al., 2021). This approach may advance the identification of linguistic form, but might ultimately have little to say about the relation of meaning to the social world (Bender and Koller, 2020; Bisk et al., 2020). Secondly, even with more modest ambitions, several NLP applications—e.g., with
Human Language Technologies, pages 4403 - 4417
spatial (McKenzie and Adams, 2021) or historical (Kutuzov et al., 2018) data—require that linguistic patterns be related to other types of structure. Thirdly, from sociological and sociolinguistic perspectives, meaning intrinsically varies as language is used in different settings and indexed to different social categories (Labov, 1972; Bourdieu, 1984; Silverstein, 2003; Eckert, 2008; Hovy, 2018). Finally, without a broader sense of context, NLP and language modeling in particular remains trapped in a paradigm where language is always treated as universal, making invisible both different speech communities (e.g. Nguyen et al., 2021) and the biases of language (e.g. Blodgett et al., 2020; Lu et al., 2020; Sap et al., 2020).
After a brief history of distributional semantics, we outline Harris' and Firth's research on distribution and, more broadly, on the scope of linguistic analysis. We look at some of the ways in which NLP has tried to account for broader context within the distributional paradigm. We suggest that existing strategies can be understood in terms of either "comparative stratification" or "syntagmatic extension." We conclude with thoughts on why rereading Harris and, in particular, Firth might aid the field of NLP with its current aporias. If words shall be known by the company they keep, then the question follows: what kind of company do they keep? Are they found only alongside linguistic elements, or do they mingle with other types of entities? Or, as Firth himself wrote: "Many different answers could be given to the question 'Distribution of what, where and how?'" (Firth, 1957a, v).
2 Background: Distributional Semantics and NLP
Distributional semantics has been an fundamental part of computational linguistics since the beginnings of the field, but in a discontinuous manner encompassing at least two distinct eras. Firstly, during the 1950s and 60s, Harris was integral to the mathematization of linguistics in the US after the Second World War (Rubenstein and Goodenough, 1965; Léon, 2021). Firth was skeptical of efforts to mechanize linguistics,<sup>1</sup> but he nonetheless consulted for some of the early work on machine translation at Cambridge Language Research Group
(Léon, 2007, 410), which included Firth's pupil, M.A.K. Halliday (Léon, 2021, 144) and shortly later the NLP pioneer Karen Sparck Jones (Léon, 2021, 89). Naturally, others also contributed to this first wave of distributional thinking, including Shannon (1945; 1948) with what might be considered one of the first language models, Warren Weaver (1952) with an early proposal for distributional semantics, and Martin Joos (1950) with a statistical formulation of language as a symbolic system of conditional probabilities.
Secondly, when computational linguistics returned to its "empiricist" roots in probabilistic methods and information theory in the mid-80s and early 90s (Norvig 2012; Léon 2021, 141), Firth and Harris accompanied Shannon among the authors who were invoked, in an ACL "Special Issue on Computational Linguistics Using Large Corpora," as foundational figures of a tradition that had been overshadowed for decades by the "rationalism" of characters such as Noam Chomsky and Marvin Minsky (Church and Mercer, 1993, 15). During this "corpus turn," the rapid automation of linguistics was driven by a resumed connection with postwar computational linguistics and information theory (Léon, 2021, 3). However, the 1990s wave of vector semantics papers that used methods like singular-value decomposition (SVD) to produce early "dense vector" models of meaning like LSA (Deerwester et al., 1989, 1990; Landauer and Dumais, 1997) and its derivatives (Hofmann, 1999; Blei et al., 2002), HAL (Burgess, 1998), or the models of Schütze (Schütze, 1992, 1993; Schütze and Pedersen, 1993) generally did not cite Firth or Harris, although a few papers from that period did (Church and Hanks, 1989; Hindle, 1990). In short, while Firth and Harris were not regularly used as stand-ins for linguistic theory during the 1990s and early 2000s, a general revival of empiricism and distributional approaches to meaning signaled a potential resurgence of interest in their thinking.
During the 2000s, the application of neural networks to language modeling tasks (e.g. Bengio et al., 2003) and the development of self-supervision techniques (e.g. Raina et al., 2007) set the stage for the word embedding breakthroughs of the early 2010s (e.g. Mikolov et al., 2013). By the end of the decade, the introduction of attention (Graves et al., 2013; Bahdanau et al., 2015) and then of the Transformer model (Vaswani et al., 2017) made way for the next breakthrough, the
<sup>1</sup>He seemed to consider the idea Orwellian (Firth, 1957b) and repeatedly attacked Norbert Wiener (e.g. Firth, 1968a,c), a pioneer whose work would later be considered foundational for connectionist AI (Goodfellow et al., 2016; Russell and Norvig, 2020).
large-language modeling revolution (e.g. Peters et al., 2018; Devlin et al., 2019; Chowdhery et al., 2022).
Following the introduction of the word2vec model and its powerful but "static" embeddings, Harris in particular was frequently cited (Le and Mikolov, 2014; Levy and Goldberg, 2014; Levy et al., 2015; Bojanowski et al., 2017), often (but not always) along with Firth (Bruni et al., 2014; Hamilton et al., 2016; Goldberg, 2017; Eisenstein, 2019; Jurafsky and Martin, 2021). However, despite an explosion of citations (Bisk et al., 2020, 8719), this interest has not been very engaged. In fact, the canonization of Firth and Harris during this time is paradoxical. On the one hand, it seems that they are invoked to lend theoretical authority to a field that struggles to lift its gaze from the latest state-of-the-art numbers (Manning, 2015; Bender and Koller, 2020). Yet, the unspoken conclusion from the ascent of neural models and the language modeling revolution was that "learning from data made linguistic theories irrelevant" (Henderson, 2020, 6295). In other words, just as NLP seemed to lose interest with linguistic theory, it elevated two pioneering theoreticians to canonical status, but seemingly without engaging closely with their work. In fact, it often seems as if Firth and Harris are referenced in such a cavalier manner that it deflects attention from the field's general lack of engagement with linguistic theory. Meanwhile, Firth and Harris became figures who justify a relatively narrow conception of meaning, one that is predominantly intra-linguistic, without much to say about its usage in social life.
This peculiar story has not been properly told. Though Léon discusses the contrast between Harris and Firth in the context of corpus linguistics (2008) and their influence on the history of computational linguistics (2021), her work does not address the differences in their distributional theories and conceptions of "context," nor the renewed and paradoxical significance of the two authors for language modeling. In our contribution, we emphasize the gap between the ideas of Firth and Harris as well as the insights a re-reading of their work offers for expanding the scope of computational semantics.
3 Harris's distributional structuralism
Few linguists contributed more to linguistic theory than Zellig Harris (1909-1992), and not just
by serving as Noam Chomsky's doctoral advisor. In fact, the two came to share little in common (Goldsmith, 2005; Nevin, 2010). Whereas Chomsky's generative grammar repositioned linguistics as a cognitive science seeking to understand, in so few words, the idealized mental representations and structures enabling language acquisition and production (e.g. Chomsky, 1972), Harris' radically distributional approach to language effectively elevated the natural language corpus as the sole starting point from which linguistic theory could arise (Harris, 1951, 1; Harris et al., 1988, 2-3; Johnson, 2002, 143-144).
This theory consisted of a linguistic structure segmentable into a finite set of formal objects characterized by constrained patterns of correspondence with one another (Harris, 1951, 1954, 1991). Such patterns of correspondence can be observed only in language-in-use, that is, in natural language corpora. In his foundational paper "Distributional Structure," for example, Harris (1954, 156-157) provides a purely distributional account of how one might induce the semantic meanings of oculist, eye-doctor, and lawyer from the partial (in the case of oculist and lawyer) or nearly complete (in the case of oculist and eye-doctor) overlap in their observed "environments" of use. This approach applies to other levels of linguistic analysis, such as morphophonemics (e.g., Harris, 1954, 155). Rather than producing a series of descriptive rules for the distribution of each phoneme, morpheme, or word, greater parsimony was sought by grouping these elements into structurally equivalent classes—categorized by their relationships as "operators" and "arguments" in Harris' later work (e.g. 1968; 1988; 1991)—sharing the same distributional rules, compounding elements in a hierarchical manner.
In this section we draw attention to three aspects of Harris's distributional linguistics: the relationship it posits between meaning and form; assumptions about heterogeneity among speakers of the same dialect; and the concept of sublanguages.
3.1 Meaning and form
A result of Harris' vision of a linguistics—concerned above all with the structural and probabilistic constraints governing the combination of formal elements—is that the discipline would be fully autonomous, not only from biology and psychology but even from semantics, phonetics, and logic, “complete without intrusion of other features
such as history or meaning” (Harris 1954, 146; Goldsmith 2005, 725–726).² Harris’s reasoning depended on the particular status he gave to linguistics among all the sciences. Taking language as its object of inquiry, linguistics lacks—unlike other sciences—a metalanguage external to language, i.e., to its object of inquiry (Harris, 1991, 4–5; Nevin, 1993, 356). Even if some other symbolic system is used, “those symbols will have to be defined ultimately in a natural language” (Harris, 1991, 274), as the surging demand for interpretability and explainability in NLP has made evident (see e.g. Danilevsky et al., 2020). Language is consequently not a “code” of “forms” that correlate with some meanings outside of it. It has no “one-to-one” conformity “with some independently discoverable structure of meaning” (Harris, 1954, 152). Instead, it is a system related to, but also independent from, thought (Harris, 1991, 383–384; Nevin, 1993, 361–363). While all human activity is meaningful, the particular meanings of language are constituted by its form, not correlated with it (Nevin, 1993, 394). Meaning thus understood is about departures from equiprobability in the distribution of these constraints (Harris, 1991, 23). These departures “define a range of meaning for each morpheme, which includes its meaning in each occurrence”. Nonetheless, shared environments do not necessarily imply shared meanings: “bumped into a pole can be said after a minor accident or after a chance meeting with an East European” (Harris, 1951, 191).
The ultimate goal of Harris's linguistic inquiry is to evaluate the efficiency of different grammars and their ability to model the statistical constraints imposed upon the distribution of different linguistic elements (Goldsmith, 2005, 723-725). Harris held that language was a "detached pattern" (Harris, 1941, 295)—information that was public and socially transmissible and hence constitutive of new types of socially shared and conventional meaning (Harris, 1991, 342-345, 377-382; Nevin, 1993, 360, 365)—and linguistics could at best discover different incomplete grammars (Harris, 1991, 31-36). Though linguistics might provide insights about meaning and discourse, or about cultural practices, such findings would not bear directly on
linguistics per se (Goldsmith, 2005, 725-726). Indeed, while Harris acknowledged that our sense of word meaning is aided by "extra-linguistic situational information," words "beyond the immediate situation" are "on their own" (Harris, 1991, 368). However, Harris's method is not completely detached. Searching for a method to segment speech, he notes that the similarity of elements "reduces ultimately to the similarity of sound segments under repetition," implemented through "the pair test" in which native speakers are asked to discriminate between sound segments (Harris, 1954, 158-159), producing an observational primitive that is "more easily controlled than data on meaning" (e.g. Harris, 1951, 20).
3.2 Variation, or lack thereof
Harris's view of language and linguistics, isolated from the vagaries of social interaction and variation, is obviously difficult to reconcile with a sociolinguistic perspective. As Harris writes in his Structural Linguistics (1951, 9), his approach is meant to describe a homogeneous dialect, which "[i]n most cases...presents no problem, since the whole speech of the person or community shows dialectal consistency." Referencing this passage, sociolinguist and dialectology pioneer William Labov (1966/2006, 5) argues that "the inconsistency found in most New York City idiots is so great that the first alternative of Harris is impossible, and the second implausible."3 In other words, even at the level of the speaker, Harris's idealized, unvarying idiolect does not hold up to empirical scrutiny. Rigorous consideration of factors that Harris would deem extra-linguistic (class, race, interactional roles, etc.) are indeed essential to produce a systematic description of linguistic structure (Labov, 1972). From the sociolinguistic perspective, Harris's vision of a linguistic science fully isolated from the "intrusion" of non-verbal social life would never obtain the systemicity to which it aspired.
This sociolinguistic critique highlights fundamental limitations of Harris's perspective. Language is viewed primarily through the distributional restrictions imposed by convention, rather than by "stylistic practice" and the ways in which speakers "make social-semiotic moves, reinterpreting variables and combining and recombining them in a continual process" (Eckert, 2012, 94). We can
<sup>3</sup>New York City is not unique in this regard; it was merely the location of Labov's early pathbreaking work.
$^{2}$ Indeed, as Jacqueline Léon notes (personal communication, April 18, 2022), even calling Harris' approach "semantics" is bit of an oxymoron. We elide a full discussion on the term since "distributional semantics" has become a commonplace phrase in NLP.
study changes in discourse, as Harris himself did in an impressive volume on structures in immunological theory over time (Harris et al., 1988), but not how people make those changes, or indeed the way in which it is "the variation itself that is systemic" (Deleuze and Guattari, 1987, 93).
3.3 Sublanguages
As noted above, Harris' revival in the 1990s was driven by the new interest in "corpus linguistics" of large corpora, a research paradigm that was partly derived from Harris' notion of "sublanguages" (Léon, 2021). Harris introduced sublanguages in his book Mathematical Structures of Language, defined as "[c]ertain proper subsets of a language [which] may be closed under some or all of the operations defined in the language, and [which] thus constitute a sublanguage of it" (Harris, 1968, 152). A sublanguage is a set of sentences which are a subset of the sentences of the "whole" language. However, the grammatical constraints of the sublanguage are not necessarily those of the whole language; rather, their grammars intersect (Harris 1968, Ch 11; Kittredge and Lehrberger 1982, 1). In application the term has come to refer primarily to the grammar and vocabulary unique to or characteristic of a particular professional or scientific field (e.g. Harris, 1988), an influential concept for early information retrieval research (Sager, 1975, 1981).4 Harris believed that sublanguages could be neatly identified using the distributional methods of his general linguistic program.
4 Firth's contextual semantics
Something similar to the sociolinguistic critique of Harris could be articulated from a different perspective, namely, through the work of J.R. Firth. Firth (1890-1960)—professor of General Linguistics at the University College of London and the first holder of a chair in that subject in Britain—independently formulated a distributional theory of lexical semantics. However, unlike Harris, Firth refused to treat meaning separately from pragmatics, and words apart from their broader “context of situation.” (Robins, 1997, 205-208)
Firth never published a fully articulated exposition of his general theory of language (Robins 1997, 216; Thomas 2011, 180) and today, all of
his work is not only out of print but also mostly unavailable online. Not understood by "the contemporary scientism" of American descriptivist linguistics and its pioneers like Harris, Firth was mostly ignored on the other side of the Atlantic (Palmer 1968, 2; Pandit 1970, 280). Unlike many of his American contemporaries, Firth did not draw mainly from cognitive psychology and logic—the latter of which Firth thought had "taken the heart out of language" (Firth, 1957a, 186)—but from the work of Polish anthropologist Bronisław Malinowski (Robins, 1997, 211). Here, we focus on the evolution of his thoughts on meaning and collocation, as well as his notions of context of situation and restricted language.
4.1 Meaning by collocation
To Firth, the purpose of linguistics is to "study meaning in its own terms" (Firth 1968b, 145; Senis 2015, 289). The famous phrase about the company that words keep concerned a particular "mode of meaning": "meaning by collocation" (Firth, 1957b, 194). Anticipating vocabulary now ubiquitous in NLP, Firth thought that this level of meaning could be found by examining the "habitual collocations" of words and the "word-material" in which they are "most characteristically embedded" (Firth, 1957c, 11-12). Meaning by collocation was an abstraction of syntagmatic relations (Oyelaran 1967, 444) that went beyond "mere juxtaposition," stating instead "an order of mutual expectancy" and "mutual prehension" (Firth, 1957c, 12). While mutual expectancy could be understood similarly to Joos's (1950) conditional probabilities of occurrence or the concept of Pointwise Mutual Information (Fano, 1961), the notion of "prehension" originates in the work of philosopher and mathematician Alfred North Whitehead (1938, 1957; see also Butt 2013) and concerns the manner in which one entity grasps another and makes it part of its own experience (Christian 1959, 12; Bryant 2011, 136).
Drawing on Whitehead's "modes of thought" (1938; see also Butt 2001, 1812, Butt 2019, 28), Firth advocated a type of "polysystemic" linguistic analysis that was interested in different, congruent modes of meaning, whether phonetic, phonological, syntactic, or semantic, but always situated in broader social context (Firth 1957c, 27, 30; Robins 1997, 214). In stark contrast to Harris, Firth explicitly rejected any efforts to create "unity in linguistics" (Firth, 1968d, 48) or one system of analysis.
<sup>4</sup>An important early figure in NLP, Sager received her PhD in Linguistics at the University of Pennsylvania and was directly influenced by Harris's work. See, e.g., Hirschman, Grishman, and Sager (1975).
Citing the later Wittgenstein (1953, Firth 1957c, 11), Firth was mainly interested in the concrete use of language, reversing the schema of Ferdinand de Saussure (1916/2011) in which language (la langue) is a system "external to and on a different plane from individual phenomena," including the concrete instances (la parole) of language use (Firth 1949, 400; Firth 1950, 44-45). While his final ideas matured considerably later, Firth initially articulated his ideas about semantics in two 1935 papers, one on semantics (Firth, 1935a) and one on phonology (Firth, 1935b), using the term "contextual distribution" in both. However, Firth (1957c, 18) ultimately disavowed this initial distributional theory—which was not too dissimilar from Harris's—as "useful" but inadequate to act as the "main principle" in a theory "of structures involving the statement of the values of the elements of structure by reference to systems."