SlowGuess/NAACL_2022_OCR huggingface dataset:
By distinguishing between system (syntagm) and structure (paradigm), Firth wanted to highlight two operational principles necessary for meaning by collocation: 1) substitution within "the same level of abstraction," and 2) commutation across different levels (Robins, 1953, 140). Only substitution that does not produce commutation in a sequence, indicates similarity of value or function (Firth 1957c, 5; Firth 1968c, 23) $^{5}$ . Two words are only substitutable—and hence similar in function and meaning—if their values do not commute across a particular sentence. Substitutability, then, does not equal synonymy. Take, for example, the following two phrases containing a) prepositional and b) adverbial uses of the word "by":
(a) They go by night.
(b) They go by night after night.
Now, "by" could be replaced by the word "past" without commuting the meaning of the other words in (b). However, replacing "by" with "past" would commute with the rest of (a) in an impossible way (Firth, 1968c, 23-24). This demonstrates how substitution concerns the relationship between "by" and "past" as two elements at the same level of analysis—i.e., lexical units—but in order to account for commutation, we need to look beyond this level to other levels of abstraction.
Firth's conception of collocation and his frequent nods to Whitehead were part of his "monistic" approach that rejected the division between mind and body (Firth 1957c, 2; Palmer 1968, 5) and all the other dualities—language and thought, word and idea, signifier and signified, expression and content (Firth, 1951, 86)—that characterized the structuralist linguistics of his time. He similarly rejected any notion of linguistics as "a theory of universals for general linguistic description" (Firth, 1957c, 21). Anticipating contemporary concerns about language diversity in NLP (e.g. Bender, 2019), Firth called for the Western scholar to "de-Europeanize himself" and the English scholar, due to the universal use of his language, to "de-Anglicize himself" (Firth 1968a, 96; Senis 2015, 274).
4.2 Context and connection
Diverging from structuralist linguistics, Firth suggested that a text should always be given a "renewal of connection with experience" (Firth, 1957c, 29). This notion of meaning was influenced by Malinowski, for whom Firth worked as an assistant early in his career (Plug, 2008, 346) and from whom he borrowed the notion of "context of situation" (Firth 1935a, Robins 1997, 211). In Malinowski's view, meaning was more than just a dyadic relationship between a word and its referent, "a multidimensional and functional set of relations between the word in its sentence and the context of its occurrence" (Robins, 1971, 35). However, while Malinowski's view on meaning was entirely functional and hyper-local, Firth employed the notion of "context of situation" as a necessary abstraction, not as a shorthand for things in themselves (Firth 1950, 43, Palmer 1968, 6). Context of situation is derived from an analytical choice, "a set of categories in ordered relations abstracted from the life of man in the flux of events, from personality in society," (Firth, 1957c, 30) prehending something of importance and bracketing the rest. It is, then, not necessarily about restricting the meaning of every utterance to a specific time and place, but about defining "an abstract set of semantically relevant categories, abstracted from multitudes of actual situations, to which unique particulars could be referred." (Robins, 1971, 41-42) Firth called for the linguist to focus on "attested language text duly recorded", accounting for a text's associated context of situation and its interior relations. (Firth,
large, a language model like BERT does not account for the context of situation.
For further details, see examples provided by Bursill-Hall (1960).
A useful analogy might be the way in which BERT handles different aspects of language at different layers of the model (Tenney et al., 2019, e.g.). However, no matter how
1957c, 29-30)
Firth was famously opaque with the exact operationalization of his concepts, including context of situation, but he did provide a detailed list of the different contextual elements that a linguist should bring into relation during analysis (Firth, 1950, 43). These include the relevant features of participants (persons, personalities); their verbal and non-verbal actions; the relevant objects; and the effects of verbal action.
During Firth's lifetime, the most thorough work that put his notion of context of situation to work was an ethnographic study by his student T.F. Mitchell in former Cyreneica (today Libya) on the language of buying and selling at the local markets of different cities and villages in the region. For Mitchell (1957, 32-33), contexts that might "correlate" with particular types of text included: the spatio-temporal situation of persons in the context; the activities of participants; the attitudes of the participants; their "personalities" such as specific trade of profession, geographical and class origins, educational standard, inter-relationship, and so on.
It is worth noting that both Mitchell and M.A.K. Halliday—Firth's student who synthesized much of his theory—used words such as "correlation," "inference," and "prediction" to describe the relationship between a text and its situational context, implying that a statistical extension of their approaches would not be completely unreasonable. In fact, Halliday himself suggested as much, when he in the early 1990s made efforts to bridge his branch of linguistics with the nascent field of corpus linguistics (Halliday, 1991).
In conclusion: Firth's famous quote itself refers to collocation, while his notion of "context" implies something much broader, "the whole conceptual meaning" (Firth, 1957c, 11). Context is the ground against which the figure of the text must be understood, no matter (per Harris) how "detached" its pattern might be (e.g. Auer, 1996). Without context, collocation captures only one narrow "mode of meaning."
4.3 Restricted languages
Like Harris, Firth's revival in connection with 1990s corpus linguistics was related to his attempts to respond to practical needs of empirical research. Expanding upon his functional understanding of language, Firth developed his notion of "restricted languages" in the 1950s (Léon 2007, 7). In a
posthumously published essay, he describes social actors as "collect[ing] a varied repertory of interlocking roles" corresponding to a "constellation of restricted languages" (Firth, 1968e, 207). As people shift between locally contextualized roles, they draw upon their "repertory" of restricted languages with specialized vocabulary and discursive styles that both reflect and constitute these contexts. Thus one might speak of a "restricted language of science, sport, defense, industry, aviation, military services, commerce, law and civil administration, politics, literature, etc." (Léon, 2008, 261). As such, the concept of restricted language is now generally seen as a precursor to the concept of "register," which was taken up by subsequent sociolinguists and linguistic anthropologists (e.g. Halliday, 1968; Gumperz and Hymes, 1972; Agha, 2005).
In proposing restricted languages as the proper object of descriptive linguistic analysis, Firth was making a broader theoretical point against, on the one hand, "the monosystemic view of language" of neo-Bloomfieldians like Harris and "pointless discussions on metalanguage" on the other, for metalanguage could be reanalyzed as a "restricted language of linguistics" itself (Léon, 2007, 9). Simply put, a descriptive linguistics which privileges restricted languages also necessarily privileges contexts of situation as an essential dimension of variation that allows social meaning to inhere in language.
5 Discussion: Words in mixed company
Often cited, together or separately, to justify a distributional approach to semantics, Firth and Harris nonetheless offer differing views on language and meaning. Harris offers us a rigorous formalism that treats language as a “detached pattern”—not a “code”, but a particular system of meaning. Firth, by comparison, left a much more scattered legacy that was only systematized by his students. Firth and Harris shared a concern about the lack of an external metalanguage of linguistics, but drew different conclusions from it. If Harris responded to this conundrum by creating one hierarchically organized system without intrusion from extra-linguistic factors, Firth called for an investigation of language as a “spectrum” (Firth, 1951, 76) with different modes of meaning that had to be addressed through multiple levels of analysis—starting with the context of situation and proceeding from there to decide which other levels are
relevant (Firth, 1950, 44). Firth's distributional theory has been described unfavourably as based on frequent co-occurrence, in contrast to the recursive dependencies developed by Harris (Habert and Zweigenbaum, 2002, 205). For Harris, the meaning of a word depends on its set (e.g. Harris, 1991, 17) such that, for example, the words "divide" and "multiply" operate on the word "cell" (and vice versa) in the same way, producing essentially the same meaning (Harris 1988, 62). However, Firth's final method of substitution and commutation also establishes complex, multidimensional criteria for distributional contrast as well as a framework for understanding polysemy. Though less formalized and less obviously recursive than Harris', Firth's approach can, arguably, also be read as treating linguistic elements as operators and arguments defined by their sets in a complex hierarchy (Firth, 1950, 44; Firth, 1951, 76 "at a series of congruent levels" (Firth, 1957c, 29-30) with different "bands of abstraction" (Firth, 1968d, 49), including the extra-linguistic context of situation. Harris and Firth both understand any linguistic analysis as incomplete, for Harris always a pursuit of the least description" (Harris, 1988, 3)—i.e., best "grammar" or model—for Firth always grounded in the social construction of facts, without any possibility of "complete axiomatization" (Firth, 1968d, 44-45).
From the perspective of empirical work, especially decades after their time, Firth and Harris also share similarities. Both rejected the mentalism that was so prevalent during their time. They were both revived as empiricist originators during the rise of statistical learning in the 1990s, and their respective work on restricted languages and sublanguages largely conflated in service of the practical concerns of corpus linguistics (Léon, 2008). Their theories both included in what we might call a "reality principle," a final arbiter of meaning outside of form: the pair test for Harris, the context of situation for Firth. The former grounds linguistics in the smallest possible unit of analysis as understood by the native speaker, the latter in social actions and objects.
In light of recent calls to extend the "world scope" (Bisk et al., 2020) of NLP and to move towards pragmatic notions of meaning, it might make sense to balance Harris's formalism and Firth's pluralism. Though Firth warned us against overextending linguistics, he was generous with the company that words could keep. They mingled with each other,
but also with events, objects, people, and indexical features such as time and space. And if NLP is ready to move beyond the corpus, then even Harris might acknowledge that when modeling language in "the immediate situation"—whether in online interactions or face-to-face communication—words are not on their own, that to judge the meaning of a combination of words, we can summon "the aid of some of the extra-linguistic situational information" (Harris, 1991, 368). In the following subsections, we consider two ways in which NLP is already doing this, in order to highlight some already existing strategies for broader contextualization. We call these strategies "comparative stratification" and "syntagmatic extension."
5.1 Comparative stratification
Corpus linguistics emerges from the question of what kind of company words keep, depending on their context. The issue of context was motivated by the introduction of corpus linguistics both for students of Firth—who considered restricted languages as a way of handling context—and for Harris's sublanguages, which were "contextually situated and suitable for being processed automatically" (Léon, 2021, 149–150). However, beyond just studying the restricted corpus, we might also consider the ways in which large datasets can be "stratified," systematically dividing them into subcorpora that are studied in relation to each other. Here, the company that words keep among each other is limited for analytical purposes, but in a manner that implies a relationship between that "company" and situational context.
Diachronic embeddings are especially representative of this approach. By stratifying timestamped data into a number of intervals, training separate models for each and then aligning the embeddings using either "second-order embeddings" or methods such as linear transformations (Kutuzov et al., 2018), the analyst can effectively represent a temporal "context of situation." This approach works with both static (Hamilton et al., 2016) and contextual embeddings (Martinc et al., 2020). While variants of this approach are most commonly used to study semantic shifts (e.g., Garg et al., 2018; Kozlowski et al., 2019; Mendelsohn et al., 2020), it could plausibly be used to stratify a dataset according to other variables such as space (e.g. Bamman et al., 2014; Gong et al., 2020), online communities (Lucy and Bamman, 2021), persons (Yao et al.,
2020), or domains (Spinde et al., 2021). Words still only keep the company of one another, but by limiting their company we implicitly introduce other participants in the analysis.<sup>7</sup>
5.2 Syntagmatic extension
Recall that for Firth, meaning by collocation and considerations of a "typical context of situation" (Firth, 1950, 44) were exercises in abstraction, with collocation being an abstraction at the syntagmatic level. Instead of restricting the company words keep, we might follow Firth's recommendation to consider them in wider company "of the same abstract nature" (Firth, 1950, 7). In vector semantics, this would imply that we explicitly introduce different contextual factors in the same vector space with our words, endowing them all with ontological equality.
The paragraph vectors introduced by Le and Mikolov (2014) as an extension of the earlier Skipgram algorithm (Mikolov et al., 2013) are representative for this approach. In practice, this method extends the syntagmatic chain of words by introducing a vector for the document as a new paradigmatic element. In principle, this type of "global context" (Grbovic and Cheng, 2018) could be anything and include several paradigmatic elements, as we can see in the research on multi-modal embeddings (e.g. Baroni, 2016) and generative modeling (e.g. Ramesh et al., 2022). Models have been developed that include demographic (Garimella et al., 2017) or persona (Li et al., 2016) vectors in the embedding space, such that intra-textual relations are accompanied by information about speakers' social categories. However, implemented with static embeddings and without some additional grammar restrictions, these context vectors essentially add only a "bag of contexts." For static embeddings, additional grammar constraints could be introduced, as was done in research on Point-of-Interest (POI) data in the domain of geoseantics, where researchers constrained contextual vectors using spatial variograms (Yan et al., 2017). Beyond static embeddings, large-language models and their dynamic embeddings could either be pretrained (with the appropriate dataset) or finetuned on data with
text associated with different contextual variables. This would realize the proposal that Halliday made in the early 1990s when he suggested an extension of the language modeling schema from the early work of Shannon, to a model with "global probabilities, those of the grammar of English, and the locally conditioned probabilities, those of this or that particular register" (Halliday, 1991, 37).
This paper revisited the theories of the two most well-known progenitors of the distributional approach to meaning in NLP. Recognizing the open question of how to bring NLP beyond the corpus, we offer a thorough account of the two distributional theories that are most often invoked to justify the modeling of meaning through departures from randomness in the company that words keep. Comparing the work of Harris and Firth—who both published their major work before the rise of the internet and its corpora—we find two distinct theories of distribution: one formal and mathematical, treating language as a particular type of detached information, another more schematic and anthropological, treating language as a functional spectrum which always emanates from a particular context of situation. The legacies of both Firth and Harris can be seen in the current paradigm of corpus linguistics, but in the domain of distributional semantics, it is Harris's ethos that dominates, despite Firth providing its most famous tagline.
Moving forward, we suggest that semantic modeling take more inspiration from Firth, and consider the context of situation and the wide variety of company that words can keep as crucial sites of innovation for the field. Doing so may not involve following a finite set of steps or flowchart. Rather, we humbly suggest that the field may be enriched by thoughtful and creative re-engagements with the intellectual traditions from which it has historically drawn. This does not imply abandoning the rigor provided by Harris. On the contrary, we find that Firth and Harris would probably have agreed that any model or "grammar" is inevitably incomplete and partial. No universal model is possible, despite the large-language modeling fervor, nor will there be one theory of language to guide us. There are only the partial perspectives and the inevitable choice of adapting one.
<sup>7</sup>In a very broad sense, the trend of pretraining large language models and then finetuning them on specific datasets is of course also an admission of the importance of “context of situation.”
$^{8}$ Firth himself (1957c, 26) recommended accompanying word definitions and collocational information with pictures.
We would like to thank Dan Jurafsky and Jacqueline Léon for their gracious and helpful comments which greatly improved the paper. Mikael Brunila is funded by the Kone Foundation, and Jack LaViolette is funded by the Mellon Foundation. We thank them for their generous support.
Asif Agha. 2005. Voice, footing, enregisterment. Journal of Linguistic Anthropology, 15(1):38-59.
Peter Auer. 1996. From context to contextualization. Links & Letters, 3(1):11-28.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, Conference Track Proceedings, pages 1-15, San Diego, CA, USA.
David Bamman, Chris Dyer, and Noah A. Smith. 2014. Distributed representations of geographically situated language. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 828-834, Baltimore, MD, USA. Association for Computational Linguistics.
Marco Baroni. 2016. Grounding distributional semantics in the visual world. Language and Linguistics Compass, 10(1):3-13.
Emily M. Bender. 2019. The #BenderRule: On naming the languages we study and why it matters. *The Gradient*.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pages 610-623, New York, NY, USA. Association for Computing Machinery.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137-1155.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr
Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online. Association for Computational Linguistics.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2002. Latent Dirichlet Allocation. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 601-608. MIT Press.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Pierre Bourdieu. 1984. Distinction: A social critique of the judgement of taste. Harvard University Press, Cambridge, MA.
E. Bruni, N. K. Tran, and M. Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1-47.
Levi R. Bryant. 2011. The democracy of objects. Open Humanities Press, Ann Arbor, MI, USA.
Curt Burgess. 1998. From simple associations to the building blocks of language: Modeling meaning in memory with the HAL model. Behavior Research Methods, Instruments, & Computers, 30(2):188-198.
G. L. Bursill-Hall. 1960. Levels analysis: J. R. Firth's theories of linguistic analysis. Canadian Journal of Linguistics/Revue canadienne de linguistique, 6(2):124-135.
David G. Butt. 2001. Firth, Halliday and the development of systemic functional theory. In Sylvain Auroux, E. F. K. Koerner, Hans-Josef Niederehe, and Kees Versteegh, editors, History of the language sciences, volume 2 of Handbucher zur Sprach- und Kommunikationswissenschaft / Handbooks of Linguistics and Communication Science (HSK), pages 1806-1838. De Gruyter, Berlin.
David G. Butt. 2013. Whiteheadian and functional linguistics. In Michael Weber, editor, Handbook of Whiteheadian Process Thought, pages 21-32. De Gruyter, Berlin.
David G. Butt. 2019. Firth and the origins of systemic functional linguistics: Process, pragma, and polysystem. In David Schonthal, Geoff Thompson, Lise