Theory-Grounded Computational Text Analysis
Year: 2023

Abstract:
In this position paper, we argue that computational text analysis lacks and requires organizing principles. A broad space separates its two constituent disciplines—natural language processing and social science—which has to date been sidestepped rather than filled by applying increasingly complex computational models to problems in social science research. We contrast descriptive and integrative findings, and our review of approximately 60 papers on computational text analysis reveals that those from *ACL venues are typically descriptive. The lack of theory began at the area’s inception and has, over the decades, grown more important and challenging. A return to theoretically grounded research questions will propel the area from both theoretical and methodological points of view.

heading: 1 Introduction
text: Computational text analysis methods—an umbrella combining natural language processing with social science—are in a honeymoon period (Lazer and Radford, 2017; van Atteveldt and Peng, 2018). Today’s social scientist might reach for the tools of computer science for their speed, scale, granularity, and consistency; for instance, natural language processing offers “to analyze signals ranging from simple lexical cues to word clusters to choices of syntactic structure” (Boydstun et al., 2014). The numerical outputs tell a story that is simple, easy to make sense of, and in that regard comforting. Conversely, today’s computer scientist may see the problems of social science as answerable by objectivity and reductionism, eschewing interpretation for quantitative analysis.
The conclusion of this reasoning, and the dominant stance in computational social science, is a reliance on machines alone to answer questions in the field, surrendering to their supposed objectivity
* Equal contribution.
or impartiality. Can a machine’s output go beyond descriptive catalogs of evidence, accelerating understanding of processes and motivations? From our experience, computers are nowhere near supplanting humans in interpreting social science results.1
An interdisciplinary inquiry must go farther than matching computational techniques to social science questions (O’Connor et al., 2011; Nguyen et al., 2020). It embraces synergistic methodology and connects the norms and standards of evidence from both. This means partnering computer science’s preference for the structured, generalizable, and objective with the unstructured, critical, and contextual which the social sciences champion. This level of interdisciplinarity addresses the question raised by descriptive findings: So what?
We see theory as the solution, empowering rather than shackling investigations. What this paper advocates is not one particular theory—certainly these are myriad, and “even subject matter which has been under intensive and prolonged study remains at the unsettled periphery of research” (Nagel, 1963). Instead, we expand on our prior work (Dore and McCarthy, 2022) to clarify calls echoed for decades by computational and social science (McDermott, 1976; Jelinek, 2005; Hajič and Hajičová, 2007; Hofman et al., 2018; Lipton and Steinhardt, 2019; Baden et al., 2021). Underlying each, we find, is the urge to return to theory, which we espouse herein.

heading: 2 Description vs. Integration
text: We contrast descriptive findings and theoretical analysis. An example of a descriptive finding is that an apple falls, or that it falls faster when pushed than dropped, or even that it falls at a particular rate estimated with some standard error by a complex
1See, e.g., Noam Chomsky’s remark on GPT-3: “You can’t go to a physics conference and say: I’ve got a great theory. It accounts for everything and is so simple it can be captured in two words: ‘Anything goes.’ All known and unknown laws of nature are accommodated. . . Of course, everything impossible is accommodated also. That’s GPT-3.” [link]
1586
interpolation. A theoretical analysis of the same phenomenon, credited to Newton, is that a fundamental force acts upon the apple, and that this same force governs the motion of the heavens. The theoretical analysis links the finding about the world critically to a broader body of knowledge and context.
Despite advances in causal inference in NLP, the descriptive is all that a machine can provide to the social sciences (Feder et al., 2021). Certainly the methods of computational text analysis have advanced since the General Inquirer (Stone and Hunt, 1963) and Mosteller and Wallace’s statistical inference of text authorship (1963). But methods are means, not ends. They uncover more descriptive findings in data: the rate of an apple’s fall, the topics of refugees’ tweets (Walk et al., 2022), the space given to marginalized groups in textbooks (Lucy et al., 2020), or patterns of state censorship (Bamman et al., 2012; King et al., 2013).
The foils to descriptive findings are integrative findings (Hofman et al., 2021), which offer causal explanations that enable future predictions—a theory, or as a ‘model’ in the sense of the Standard Model, rather than of a statistical model. Integrative findings can either offer new theories or couch their explanations in existing theories—but the theory is essential either way.
3 We Don’t Integrate
To contrast descriptive and integrative findings, we reviewed approximately 60 papers in computational text analysis published in *ACL venues. In Table 1, we describe several of these in terms of their descriptive or theory-grounded contributions.2 Descriptive papers may refer to social science theories or make generalizable claims, as when Demszky et al. (2019) write, “The shooter’s race appears to play a role in topic preference: if the shooter is white, Democrats become more likely to focus on shooter’s identity,” but they do not link to the two to each other.
An excellent theory-grounded quantitative work is Nelson (2021); she confirms some of the most compelling features of identity theory, specifically that identities based on race were most distinguished by cultural discourse, whereas those based on gender by the domestic and the economic discourse. Similarly, we conducted theory-grounded quantitative work to investigate the application of the protest
2Following Lipton and Steinhardt (2019), we only describe papers by established researchers to “avoid singling out junior students. . . who lack the opportunity to reply symmetrically”.
paradigm and thematic framing in how westernand Hong Kong based newspapers portray protests in Hong Kong (McCarthy et al., 2021; McCarthy and Dore, 2022). Generally, it remains challenging to find computational social science papers in *ACL venues that go beyond description and prediction, advancing theory. Why is this? We believe it stemmed from the field’s “empirical turn”.3
Few remember when the meetings of ACL offered a few dozen papers, all entrenched in formalisms and linguistic theories. Arguably, 1996 was a turning point when the founders of SIGDAT held the first EMNLP at Penn under the auspices of the ACL.4 This gave a spotlight to the few but growing empiricists in the field and drew in more.
EMNLP began a half-decade of measurable reorganization the field (Anderson et al., 2012). That EMNLP remains affiliated with ACL keeps the language-focused machine learning practitioners in our tent. The slow blurring of boundaries between each *ACL conference’s expectations (Church, 2020) increases this unity. Both groups belong under this tent. But without a doubt, one group’s voice is becoming less heard.
Publication venues within the ACL focus on methods over theory.5 Techniques are taken off the shelf without critical examination because these are “the best” (often “state of the art”) for their purposes (Ethayarajh and Jurafsky, 2020). This widens the gap between theoretical and empirical work.6 Hopkins and King (2010) claim, “computer scientists may be interested in finding the needle in the haystack. . . social scientists are more commonly interested in characterizing the haystack”—evincing the value of broader context.7 Wallach (2018), quoting Hopkins and King, explains that the two groups
3A lesser reason is the challenge of serving two masters: adequately covering both the theoretical and methodological components within 8 pages. We recently received two reviews for an *ACL submission: one advocating for more of the social science context in the main text by eschewing methods to the appendix, and the other instructing us to do the opposite.
4And its predecessor the Workshop on Very Large Corpora. 5This is due to the outsized influence of computer science, often seen as the science of method (Hoare and Jones, 1989; Shapiro, 2001), when not instead seen as an engineering discipline (Rapaport, 2005).
6A related criticism is that empirical research has narrowed to focus on ‘easy’ questions that its tools can address (Coleman, 1986; Baden et al., 2021), especially when research questions are baked into the design of the task.
7As evidence, see Siegel (2018): “We usually don’t know about causation, and we often don’t necessarily care. . . the objective is more to predict than it is to understand the world. . . It just needs to work; prediction trumps explanation.”
are interested in very different research questions, and that computational social science must be more than computer science with social data; it must strive for valid explanatory models. In the same vein, at ACL 2022, ACL fellow Eduard Hovy remarked that NLP must be more than “just machine learning on corpora”.
Social scientists are also coming to terms with the meaning of computational techniques applied more often in social science (Bail, 2014; Biernacki, 2015; Lee and Martin, 2015; Spillman, 2015). The focus of the debates, however, is on which methods are best suited to extract meaning from text, without addressing any theoretical considerations related to the methods or whether a theoretical framework for those methods even exists. The discussions on whether computational methods make social science research more efficient, reliable, and reproducible overtake attempts at theory-building.

heading: 4 Moving Forward
text: We are not denying the value of computational approaches to analyzing text. Certainly, comput-
ing can be an instrumental approach for modeling and understanding social complexity. This does not mean that other approaches, such as historical, ethnographic, or mathematical, become irrelevant. On the contrary, computational methods necessarily (whether awarely or not) rely on these earlier approaches to add value, in terms of improving our explanations and understanding (Radford and Joseph, 2020).
As we are a field that prioritizes methods, consider the seminal book on methods in science: Abbott (2004) taxonomizes scientific ways of knowing. Its five broad categories are ethnography, historical narration, standard causal analysis, small-N comparison, and formal modeling. We in NLP myopically choose the third and fifth of these, ignoring the value of the others. But the broader point of Methods of Discovery is not methods. It is the research question. Any methodology should be grounded in the question, not incremental tweaks and reviewers’ comfort (Church, 2020). This admits even qualitative or mixed-method approaches to text analysis.
The role of humans in scientific inquiry is nothing
new. Using qualitative analysis to complement quantitative techniques has its roots in Achen and Snidal (1989)’s recommendation to use historical case studies as a complement to statistical research.8 Their plea was strengthened by Verba’s work in the early 1990s (Verba et al., 1993, 1995; Verba, 1996) and Tarrow (1995), who openly called for bridging qualitative and quantitative modes of research in social science. In doing so, they have enriched the field with critical methodological innovations (Gerring, 2004), benefiting from the recognition that “quantitative methods must augment humans, not replace them” (Grimmer and Stewart, 2013, 4).
The field can draw more from social science’s rich tradition of inductive theory-building and interpretation to develop its theoretical approach—to prize either induction or deduction alone is a myth of scientific procedure (Thagard, 1988), but the melding of the two opens new doors. Rather than eschewing the complexity (a criticism leveled by Baden et al., 2021), it should put complexity at the center of its ontology on the basis that there are no immutable laws in social life or optimal solutions to social problems.
Skepticism can linger toward findings not drawn from the standard practices of one’s own field; indeed, social science was long skeptical of computational contributions (Armstrong, 1967). We believe that this drives the hyperfocus on improving a few accepted methods instead of exploring more broadly. If the doorway between disciplines is only narrowly open, this reflects a lack of appreciation for each field’s ways of knowing. The disciplinary divide keeps computational researchers from embracing methods beyond standard causal analysis or formal modeling, so the interpreter-centric richness allowed by histories, ethnographies, and small-N exploration are precluded.

heading: 5 Conclusion
text: We have explained the distinction between descriptive and theoretical findings as it pertains to computational text analysis. The bulk of work we found provided vast descriptive findings, often of high quality, but not giving back to questions of theory. We offer several suggestions on how to ‘push the pendulum back’ by prioritizing theory-building or
8Expertise plays a role as well (Shing et al., 2018), which is why Mechanical Turk doesn’t fill the need for qualitative analysis. This is exemplified by Radford and Joseph (2020)’s observation of “non-expert annotators provid[ing] unreliable annotations, even after a discussion period”.
theory-affirming research questions and accepting whichever methods are best suited toward answering it—not only the familiar and entrenched ones.
We are not the first to advocate for a shift in the patterns of applying computational techniques to real-world problems. There is a steady drumbeat from voices in the field advocating careful approaches (Nagel, 1963; McDermott, 1976; Jelinek, 2005; Hajič and Hajičová, 2007; Hofman et al., 2018; Lipton and Steinhardt, 2019; Baden et al., 2021). What we see underlying all of these— those writing against ‘mathiness’ and speculation, advocating for clear evaluation over anecdotes, criticizing textual researchers’ dilution of conceptual standards, highlighting work that ties linguistic information into complex models—is an unspoken, perhaps unrealized, call for a return to theory.
Not only do we aver that incorporating theory is essential; but also, other fields have strengthened themselves when espousing organizing principles beyond those of their progenitors. Behavioral economics is a success story here. It transcended the neat (but psychosocially stripped) mathematics it draws from to acknowledge deviations from rationality and blend economics with cognitive science (Kahneman and Tversky, 1979; Thaler, 1980; Thaler and Sunstein, 2009).
For scientific—not simply engineering— advances to arise from the *ACL community, authors and reviewers alike must resist the temptation toward incremental, ‘safe’ research and follow Church (2005): “Controversial papers are great; boring unobjectionable incremental papers are not.” In reviewing new research, we should privilege not only work that presents new and unusual computational methods, but also interactions between computational and humanistic approaches to answering research questions. EMNLP was founded because of reviewing biases at ACL against groundbreaking methodological advances, and since then the two have homogenized; “EMNLP reviewing is no longer much of a differentiator” (Church, 2020). We found that theoretically grounded findings in text analysis are often published in non-*ACL venues (Table 1), but ACL sets the standard for work involving computational text analysis and NLP. Is there no home for groundbreaking integrative or interdisciplinary work in *ACL, such that a new venue is required? Or can we adapt our standards to invite deeper connections to theory and new ways of knowing?

heading: Acknowledgments
text: This publication was made possible in part by a grant from the American Political Science Association to A.D.M. and G.M.D.D. The statements made and views expressed are solely the responsibility of the authors. A.D.M. is supported by an Amazon Fellowship and a Frederick Jelinek Fellowship.

heading: Limitations
text: The key limitation of our work is that, when conducting the review of approximately 60 papers (by searching through the ACL Anthology for works in computational social science since 2010), we encountered a skewed distribution of descriptive versus integrative works. In fact, it was relatively simple to find descriptive works, and that section of Table 1 could have been much longer. We also recognize that, due to the mixed nature of our field, scientific and integrative findings are not the only goal—our ‘big tent’ includes engineers as well, who value gains in performance indicators. Finally, the fact that we have few examples of papers showing a return to theory renders the possibility that our central claim is misinterpreted in a normative way as a mandate.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?

heading: Unnumbered; appears on page 5.
text: 7 A2. Did you discuss any potential risks of your work? This is a position paper.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?

heading: Left blank.
text: 7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts?

heading: Left blank.

heading: B1. Did you cite the creators of artifacts you used?

heading: Not applicable. Left blank.

heading: B2. Did you discuss the license or terms for use and / or distribution of any artifacts?

heading: Not applicable. Left blank.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.

heading: Not applicable. Left blank.
text: C 7 Did you run computational experiments?

heading: Left blank.

heading: C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Not applicable. Left blank.

heading: C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Not applicable. Left blank.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

heading: Not applicable. Left blank.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Left blank.

heading: Future Work
text: , 2021), which offer causal explanations that enable future predictions—a theory, or as a ‘model’ in the sense of the Standard Model, rather than of a statistical model. Integrative findings can either offer new theories or couch their explanations in existing theories—but the theory is essential either way. 3 We Don’t Integrate
To contrast descriptive and integrative findings, we reviewed approximately 60 papers in computational text analysis published in *ACL venues. In Table 1, we describe several of these in terms of their descriptive or theory-grounded contributions. 2 Descriptive papers may refer to social science theories or make generalizable claims, as when Demszky et al. (2019) write, “The shooter’s race appears to play a role in topic preference: if the shooter is white, Democrats become more likely to focus on shooter’s identity,” but they do not link to the two to each other. An excellent theory-grounded quantitative work is Nelson (2021); she confirms some of the most compelling features of identity theory, specifically that identities based on race were most distinguished by cultural discourse, whereas those based on gender by the domestic and the economic discourse. Similarly, we conducted theory-grounded quantitative work to investigate the application of the protest
2Following Lipton and Steinhardt (2019), we only describe papers by established researchers to “avoid singling out junior students. who lack the opportunity to reply symmetrically”. paradigm and thematic framing in how westernand Hong Kong based newspapers portray protests in Hong Kong (McCarthy et al. , 2021; McCarthy and Dore, 2022). Generally, it remains challenging to find computational social science papers in *ACL venues that go beyond description and prediction, advancing theory. Why is this? We believe it stemmed from the field’s “empirical turn”. 3
Few remember when the meetings of ACL offered a few dozen papers, all entrenched in formalisms and linguistic theories. Arguably, 1996 was a turning point when the founders of SIGDAT held the first EMNLP at Penn under the auspices of the ACL. 4 This gave a spotlight to the few but growing empiricists in the field and drew in more. EMNLP began a half-decade of measurable reorganization the field (Anderson et al. , 2012). That EMNLP remains affiliated with ACL keeps the language-focused machine learning practitioners in our tent. The slow blurring of boundaries between each *ACL conference’s expectations (Church, 2020) increases this unity. Both groups belong under this tent. But without a doubt, one group’s voice is becoming less heard. Publication venues within the ACL focus on methods over theory. 5 Techniques are taken off the shelf without critical examination because these are “the best” (often “state of the art”) for their purposes (Ethayarajh and Jurafsky, 2020). This widens the gap between theoretical and empirical work. 6 Hopkins and King (2010) claim, “computer scientists may be interested in finding the needle in the haystack. social scientists are more commonly interested in characterizing the haystack”—evincing the value of broader context. 7 Wallach (2018), quoting Hopkins and King, explains that the two groups
3A lesser reason is the challenge of serving two masters: adequately covering both the theoretical and methodological components within 8 pages. We recently received two reviews for an *ACL submission: one advocating for more of the social science context in the main text by eschewing methods to the appendix, and the other instructing us to do the opposite. 4And its predecessor the Workshop on Very Large Corpora. 5This is due to the outsized influence of computer science, often seen as the science of method (Hoare and Jones, 1989; Shapiro, 2001), when not instead seen as an engineering discipline (Rapaport, 2005). 6A related criticism is that empirical research has narrowed to focus on ‘easy’ questions that its tools can address (Coleman, 1986; Baden et al. , 2021), especially when research questions are baked into the design of the task. 7As evidence, see Siegel (2018): “We usually don’t know about causation, and we often don’t necessarily care. the objective is more to predict than it is to understand the world. It just needs to work; prediction trumps explanation. ”
are interested in very different research questions, and that computational social science must be more than computer science with social data; it must strive for valid explanatory models. In the same vein, at ACL 2022, ACL fellow Eduard Hovy remarked that NLP must be more than “just machine learning on corpora”. Social scientists are also coming to terms with the meaning of computational techniques applied more often in social science (Bail, 2014; Biernacki, 2015; Lee and Martin, 2015; Spillman, 2015). The focus of the debates, however, is on which methods are best suited to extract meaning from text, without addressing any theoretical considerations related to the methods or whether a theoretical framework for those methods even exists