Comparing grounded theory and topic modeling: Extreme divergence or unlikely convergence?

Researchers in information science and related areas have developed various methods for analyzing textual data, such as survey responses. This article describes the application of analysis methods from two distinct fields, one method from interpretive social science and one method from statistical machine learning, to the same survey data. The results show that the two analyses produce some similar and some complementary insights about the phenomenon of interest, in this case, nonuse of social media. We compare both the processes of conducting these analyses and the results they produce to derive insights about each method's unique advantages and drawbacks, as well as the broader roles that these methods play in the respective fields where they are often used. These insights allow us to make more informed decisions about the tradeoffs in choosing different methods for analyzing textual data. Furthermore, this comparison suggests ways that such methods might be combined in novel and compelling ways. A Tale of Two Methods Text plays an important role in research, but analyzing textual data poses unique challenges. In the case of surveys, free‐text responses allow participants to report their experiences in detailed ways that might not be anticipated by researchers. However, this flexibility also creates challenges in synthesizing insights from the specificity of individual responses. This article compares two approaches to this problem, a qualitative approach from interpretive social science (Glaser & Strauss, 1967) and a quantitative approach from natural language processing (Blei, Ng, & Jordan, 2003). We demonstrate, through a running example relating to social media use and nonuse, that these methods involve surprisingly similar processes and produce surprisingly similar results. Surveys that include open‐ended, free‐text responses are often analyzed using qualitative methods (e.g., Baumer et al., 2013; Rader, Wash, & Brooks, 2012; Wang et al., 2011). Analytic methods from, e.g., grounded theory (Charmaz, 2006; Glaser & Strauss, 1967) can generate rich, thick descriptions (Geertz, 1973). Coming from interpretivist (cf. Ma, 2012; Orlikowski & Baroudi, 1991) analysis of empirical social phenomena, these approaches emphasize how a social group co‐constructs both their reality and its meaning. However, such methods are time‐ consuming and difficult to apply to massive datasets. Furthermore, concerns can emerge about the particular subject position(s), and concomitant bias(es), of the researcher (Clifford & Marcus, 1986). Methods from machine learning provide an alternative approach. Roberts, Stewart, and Tingley (2014a) analyzed open‐ended survey responses using topic models (Blei et al., 2003), which find statistical regularities in word co‐occurrence that often correspond to recognizable themes, events, or discourses (Jockers & Mimno, 2013). This method scales to billion‐word datasets and can arguably provide an analysis driven more by the documents than by human preconceptions. However, algorithmically defined “topics” run the risk of misleading researchers. Not only does properly making sense of the results require detailed technical knowledge, but these techniques forgo a certain degree of human contextual interpretive ability (Rost, Barkhuus, Cramer, & Brown, 2013). Moreover, an appeal to computational objectivity may embody fundamentally different epistemic commitments than those at work in interpretivist approaches (Ma, 2012; Orlikowski & Baroudi, 1991). Each method thus carries its own strengths and weaknesses. We can algorithmically identify latent patterns in complex datasets with relative speed and ease, but in a way that potentially forfeits an appreciation for context, subtlety, and an interpretive approach to social reality. Alternatively, we can conduct detailed analyses of particular sociotechnical practices, but the effort involved precludes analyzing data collected from larger pools of respondents. These tensions arise not only in the analysis of surveys but also for a wide variety of data sources, from usage logs and scraped data (Backstrom, Boldi, Rosa, Ugander, & Vigna, 2012; Schoenebeck, 2014) to popular media coverage (Harmon & Mazmanian, 2013; Lind & Salo, 2002; Portwood‐Stacer, 2013) to policy documents (Epstein, Roth, & Baumer, 2014). However, we have relatively little understanding of how these differences play out in practice. Mixed methods research combines different methods (e.g., Baumer et al., 2013; Schoenebeck, 2014), but such work rarely examines similarities and differences in the methods themselves. To the authors' knowledge, no empirical comparative analysis has examined how approaches from the interpretive social sciences and computational analysis techniques either converge, diverge, or both when applied to the same data. To conduct such a comparison, we consider a case study in social media reversion (Baumer, Guha, Quan, Mimno, & Gay, 2015), that is, when a social media user leaves a site to become a nonuser (Satchell & Dourish, 2009; Wyatt, 2003) but then subsequently returns to become a user again. Social media reversion represents a phenomenon that has been identified as important in prior work on technology nonuse (Baumer et al., 2013; Brubaker, Ananny, & Crawford, 2014; Schoenebeck, 2014) but has not yet received significant attention (Wyatt, 2003). To study this nascent phenomenon, we leverage data from an online campaign by the Dutch advertising firm Just that encouraged users to stay off of the social networking site Facebook for 99 days (http://99daysoffreedom.com/). Participants were then surveyed by Just after 33, 66, and 99 days. Of the more than 5,000 survey responses collected, 1,095 reported returning to Facebook before 99 days had passed. We analyzed these participants' descriptions of their experiences in returning to the site. These data provide a prime opportunity to compare qualitative and computational methods. The dataset is large enough that a computational text analysis will provide meaningful results, but it is small enough to ensure that an interpretive social scientific approach remains tractable. These data are analyzed using two separate approaches—grounded theory (Glaser & Strauss, 1967) and statistical topic modeling (Blei et al., 2003)—which were conducted independently by separate authors. We chose these particular methods due to their respective popularity among interpretive and computational approaches, their roughly analogous goals of identifying thematic patterns in unstructured text data, and their popularity among information science researchers. However, conducting a comparison of these methods purely as abstract analytic techniques would be nearly impossible. Neither topic modeling nor grounded theory are applied purely by rote. Rather, each requires nonnegligible amounts of researchers' subjective judgment, both in the application process and in interpreting results. Indeed, if two researchers conducted independent analyses following the principles of grounded theory, one would expect different results from each. Comparing these results would highlight where each analysis confirms, or perhaps discredits, the other's findings. This article takes a similar approach, but instead it compares results from a grounded theory analysis and from a topic modeling analysis. This comparison provides a better understanding of both where and how these methods for textual analysis might either converge or diverge. Our findings show both numerous areas of resonance and some key divergences. First, the authors identified several correspondences between the grounded theory themes and the algorithmically generated topics. These correspondences did not strictly map either set of results onto the other but suggest a many‐to‐many, or in most cases a “two‐to‐two,” mapping. Second, the authors found several alignments in analysis processes. Both methods begin with a provisional model or theory that is iteratively refined based on the data. Thus, the theory or model arises primarily from the data themselves. However, what is meant in each approach by the terms “model,” “theory,” “iteration,” and even “data” differ, perhaps dramatically. Third, grounded theory and topic modeling each serve important discursive functions in terms of legitimating particular analytic practices to a broader disciplinary audience. Both approaches attempt to shift the perception of particular methods on a spectrum from impressionistic to computational (Ramsay, 2003, 2011), each moving toward a potential center but from opposite directions. Thus, this article makes a primarily methodological contribution. It advances our understanding of how approaches from the interpretive social sciences and from computer science both resemble and differ from one another, both in terms of substantive results and in terms of analytic process. It also argues that these methods should be considered in the context of the broader theoretical and methodological shifts they represent in their respective fields. The article concludes by suggesting directions to orient future work that explores mixed computational‐interpretive methods.