Question: Why computer scientists may feel uncomfortable about  social and cultural concepts in analysis?
Evidence: A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities. Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them. In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research. For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights. The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses.
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13 . The background concept comprises the full and diverse set of meanings that might be associated with a particular term. This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated. That definition, in turn, represents the systematized concept: the formulation that is adopted for the study.
It is important to consider that for social and cultural concepts there is no absolute ground truth. There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time. This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e.g., by comparing a sentiment classifier's output against manual annotations. However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning. BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions". BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [..] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists." The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges. Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18 . We return to this issue in our discussions on validation and analysis.
It is important to consider that for social and cultural concepts there is no absolute ground truth. There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time. This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model’s output against “ground truth” or a “gold standard”, e.g., by comparing a sentiment classifier’s output against manual annotations.
Answer: Because there is no absolute ground truth for social and cultural concepts, and definitions might be contested over time. However, for computer scientists, their primary measure of success is often based on comparing a model’s output against “ground truth” or a “gold standard”.