It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance
Year: 2023

Abstract:
Progress in NLP is increasingly measured through benchmarks; hence, contextualizing progress requires understanding when and why practitioners may disagree about the validity of benchmarks. We develop a taxonomy of disagreement, drawing on tools from measurement modeling, and distinguish between two types of disagreement: 1) how tasks are conceptualized and 2) how measurements of model performance are operationalized. To provide evidence for our taxonomy, we conduct a metaanalysis of relevant literature to understand how NLP tasks are conceptualized, as well as a survey of practitioners about their impressions of different factors that affect benchmark validity. Our meta-analysis and survey across eight tasks, ranging from coreference resolution to question answering, uncover that tasks are generally not clearly and consistently conceptualized and benchmarks suffer from operationalization disagreements. These findings support our proposed taxonomy of disagreement. Finally, based on our taxonomy, we present a framework for constructing benchmarks and documenting their limitations.

heading: Future Work
text: 6 A Framework for NLP Benchmarks
Towards better documenting benchmarks’ conceptualization and operationalization, we encourage benchmark creators to answer the questions in Table 3 in their future directions or limitations section when they propose a new benchmark Bτ for a task τ. This framework is not a post-hoc intervention. We intend for benchmark creators to answer these questions before, during, and after they construct benchmarks; this framework should be grounded in care for and facilitating collective progress in NLP. Furthermore, creators should share their answers to these questions, so that this framework becomes normalized and shapes people’s thinking about their own contributions. Moreover, this framework is intended to supplement processes like Datasheet for Datasets and Data Statements for NLP (Gebru et al. , 2021; Bender and Friedman, 2018), which enable comprehensive documentation for benchmarks, but do not ask benchmark creators to reflect in a way that distinguishes between: 1) how they conceptualize a task (and how others may disagree with their conceptualization), and 2) how well the benchmark operationalizes a measurement model for model performance on
their conceptualization of the task. This framework is also complementary to technical solutions (e. g. , human-in-the-loop approaches) to resolving task ambiguity (Tamkin et al. , 2022). We hope that this reflection will benefit the NLP community in the following ways: • Reduce overhyping: By being transparent about
and defining the model capabilities that benchmarks are intended to assess, as well as documenting benchmark validity issues, benchmark creators will: 1) not misrepresent model capabilities, and 2) remind people to be careful about extrapolating benchmark performance results. • Encourage reflexivity and engagement with the politics of benchmarks: By clarifying how they conceptualize tasks and considering how others may disagree with their conceptualization, benchmark creators will: 1) assess how their social context and power influences task conceptualization and benchmark construction (Collins, 2017), 2) reflect on which groups of people benchmarks represent, and 3) include people from diverse communities during benchmark construction towards alleviating disagreement. Towards considering historical and social context,
we urge practitioners to not neutralize disagreements in conceptualization by valuing all “sides” equally, as this inevitably invalidates marginalized people’s lived experiences and perpetuates the power relations in which benchmark construction participates (Collins, 2017; Denton et al. , 2021). In particular, the widespread adoption, presumed validity, and inertia of benchmarks influence the direction of NLP, shaping funding landscapes and the domains in which NLP systems are deployed (Blili-Hamelin and Hancox-Li, 2022; Bommasani, 2022). As such, we encourage practitioners to prioritize the perspectives of marginalized people. • Provide actionable insights to address benchmark validity issues: Distinguishing between conceptualization and operationalization disagreements in a benchmark will better enable the creators of the benchmark, as well as creators of future benchmarks, to address benchmark validity issues. For example, to address perceptions that a benchmark does not exhaustively assess whether models can “handle realworld phenomena,” benchmark creators can decide if this is a conceptualization disagreement (e. g. , “real-world” is too open-ended, in which case creators should clearly explain which domains they foreground in their conceptualization of “real-world”) or operationalization disagreement (e. g. , acquiring real-world data is difficult