<figure><img src="image_1.png" /></figure>

HAL
open science

# The NeWMe Corpus: A gold standard corpus for the study of Word Meaning Negotiation

Aina Garí Soler, Jenny Myrendal, Chloé Clavel, Staffan Larsson

<figure><img src="image_2.png" /></figure>

## To cite this version:

Aina Garí Soler, Jenny Myrendal, Chloé Clavel, Staffan Larsson. The NeWMe Corpus: A gold standard corpus for the study of Word Meaning Negotiation. 2025. hal-05423801

HAL Id: hal-05423801

https://inria.hal.science/hal-05423801v1

Preprint submitted on 18 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The NeWMe Corpus: A gold standard corpus for the study of Word Meaning Negotiation

Aina Garí Soler

aina.gari-soler@inria.fr

INRIA Paris

Jenny Myrendal

Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg

Chloé Clavel

INRIA Paris

Staffan Larsson

Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg

## Research Article

**Keywords:** Word meaning negotiation, WMN, semantic alignment, semantic coordination, interactional linguistics, misunderstanding

**Posted Date:** April 23rd, 2025

**DOI:** https://doi.org/10.21203/rs.3.rs-5975927/v1

**License:** cc BY This work is licensed under a Creative Commons Attribution 4.0 International License.
Read Full License

**Additional Declarations:** Competing interest reported. This work was supported by the Swedish Research Council (VR) grant 2022-02125 Not Just Semantics: Word Meaning Negotiation in Social Media and Spoken Interaction, and by state funding managed by the Agence Nationale de la Recherche under the France 2030 program, with reference "ANR-23-IACL-0008". The authors have no financial or proprietary interests in any material discussed in this article.
# The NeWMe Corpus: A gold standard corpus for the study of Word Meaning Negotiation

Aina Garí Soler<fnref n="1" />,<fnref n="2" />* · Jenny Myrendal<fnref n="3" />,<fnref n="4" />* · Chloé Clavel<fnref n="1" />,<fnref n="2" /> · Staffan Larsson<fnref n="3" />

## Abstract

Word Meaning Negotiation (WMN) sequences occur when participants focus on clarifying or negotiating the meaning of a word or phrase, often prompted by questions or challenges. These interactions temporarily shift the conversation to explore nuances of meaning - sometimes resulting in quick clarification when due to insufficient understanding of word meaning, and other times leading to extended debates, such as disagreements on what a word can or should mean.

This paper presents the largest and freely available manually annotated corpus of WMNs to date, encompassing spoken dyadic and multiparty conversations as well as online discussions. Our methodology combines searching for WMNs using regular expressions with a detailed annotation scheme that categorizes WMNs into types triggered by non-understanding (NONs: Non-understanding WMN) or disagreement (DINs: Disagreement WMN), and distinguishes between negotiations of situated and potential meanings. We also annotate incomplete negotiations and related phenomena, and analyze inter-annotator agreement to evaluate the reliability of the annotation schema.

Preliminary investigations of WMNs in the corpus reveal distinct patterns in WMNs across contexts, with NONs prevalent in spoken interactions and DINs dominating online debates. This resource lays a foundation for studying semantic alignment, developing automated WMN detection, and creating adaptive dialogue systems. Our findings highlight the complexity of WMNs and provide practical insights for their identification and analysis.

**Keywords:** Word meaning negotiation, WMN, semantic alignment, semantic coordination, interactional linguistics, misunderstanding

## Acknowledgements

This work was supported by the Swedish Research Council (VR) grant 2022-02125 Not Just Semantics: Word Meaning Negotiation in Social Media and Spoken Interaction, and by state funding managed by the Agence Nationale de la Recherche under the France 2030 program, with reference "ANR-23-IACL-0008". We want to thank Matthieu Labeau and William Noble for their contributions to the discussions, Anh Ngo Ha for her feedback on the annotation guidelines, and Kaj Ailomaa for his work on the website infrastructure and GitHub repository, as well as the three annotators for their efforts in the annotation process.

<footnote n="1">¹ INRIA Paris, France</footnote>

<footnote n="2">² LTCI, Télécom-Paris, Institut Polytechnique de Paris, France</footnote>

<footnote n="3">³ Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg, Sweden</footnote>

<footnote n="4">⁴ Department of Education, Communication and Learning, University of Gothenburg, Sweden</footnote>

<footnote>* Equal contribution</footnote>
# 1. Introduction

Mutual understanding is a fundamental aspect of successful interaction, enabling participants to align their interpretations and ensure that communication flows freely without major interruptions. In any conversation, alignment between interlocutors can occur at various levels, including lexical choice (Brennan & Clark 1996), syntax (Branigan et al., 2000) and conceptual representations (Stolk et al. 2016).<fnref n="5" /> Alignment at the level of word meaning, which is the focus of this study, is both essential and sometimes challenging to achieve. When participants share a common understanding of word meanings, conversations tend to proceed more smoothly.

As Garfinkel (1967) observed with the concept of “current practical purposes,” interactions typically require just enough alignment to continue, even without full mutual understanding. Research has shown that achieving alignment at one level can support alignment at other levels (Cleland & Pickering, 2003; Pickering & Garrod, 2004). However, conceptual misalignments can sometimes remain hidden beneath apparent lexical alignment, and, if overlooked, have the potential to cause significant misunderstandings (Schober, 2005; Garí Soler et al., 2023). Despite its importance, conceptual (mis)alignment remains a relatively underexplored area, primarily due to the challenges involved in identifying it.

When misalignments are detected, they may lead to explicit efforts to clarify or resolve the issue. Such efforts often involve meta-linguistic discussions, where participants directly address and negotiate the meanings or uses of problematic words or phrases. These interactions, which we call Word Meaning Negotiation (WMN; Myrendal, 2015), play an important role in achieving alignment. For instance, if one speaker uses the word X and another speaker questions whether X is appropriate in the current context, the conversation may temporarily shift from the main topic to focusing on language, specifically the meaning of X. Through such sequences, participants engage in a negotiation process to explore and address the meaning of the word. They may contribute their individual perspectives, propose what the word should mean in the given context, or challenge existing interpretations by highlighting what the word cannot mean. This exchange allows participants to collaboratively explore and shape the boundaries and potentials of the word's meaning, whether the interaction involves addressing disagreement or clarifying its meaning in context.

The term *negotiate* is used broadly to encompass a range of actions and contributions, including explaining, problematizing, questioning, or exemplifying word meanings. Similarly, the term *word* is used here in an inclusive sense, referring not only to single word forms but also to lexicalized expressions or phrases. This broader definition recognizes that meaning negotiation can involve both individual lexical items and multi-word units that function as cohesive semantic entities in interaction.

Consider a scenario where one participant describes a colleague as *ambitious* and another responds, “What do you mean by ambitious: driven or just self-centered?” The dialogue may then unfold with each participant elaborating on their understanding of the word *ambitious* in the given context, aiming to achieve enough common ground to move on in the conversation. This sequence illustrates a typical WMN, where the meaning of a word becomes the focus of collaborative negotiation, temporarily shifting the conversation’s focus to language itself.

Word Meaning Negotiation is important for fostering conceptual and linguistic alignment, particularly in contexts where achieving mutual understanding is more complex. Misunderstandings arising from conceptual misalignments can disrupt communication, and WMNs serve as a mechanism to identify and resolve these issues. By explicitly addressing word meanings, WMNs facilitate collaborative efforts to clarify, refine, or redefine meanings, ensuring the conversation can proceed smoothly.

The importance of WMN may be especially relevant in environments where traditional cues for achieving alignment are limited or absent. For instance, in online interactions, participants often lack access to nonverbal

<footnote n="5">⁵ We take lexical alignment to be an agreement on what word to use for a given meaning, and conceptual alignment to be an agreement about what meaning to assign a given word. While conceptual alignment is the prototypical case of a WMN, lexical alignment also involves WMN since when deciding on which word to use for a meaning, one is also deciding on the meaning of that word (and the alternative).</footnote>
cues such as gestures, gaze, and intonation, which support alignment in face-to-face conversations (Babcock et al., 2014). This limitation increases the likelihood of misalignments and highlights the need for WMN as a tool to navigate such challenges.

While WMN resolves immediate misalignments in conversation, studying these processes has broader implications for understanding interactional dynamics. It offers insights into how participants collaboratively shape language and negotiate meaning in real time as part of everyday activities and interactions. By studying WMN, we can better understand the mechanisms that underpin alignment at both lexical and conceptual levels, as well as explore its potential to address controversial or ambiguous terms.

This paper outlines the process of establishing a gold standard corpus for the detection and study of WMN sequences, presenting what we believe to be the first manually annotated corpus of its kind, encompassing both spoken and written conversations. We propose and apply a methodology to locate likely WMN sequences, and provide an annotation schema with detailed instructions for the annotation of WMN and related phenomena observed in the data. The corpus, which will be made publicly available, includes expert annotations and we describe the process of training multiple annotators and calculating inter-annotator agreement (IAA).

While the study does not directly explore WMN as a mechanism of interactional alignment, it builds on qualitative studies that have examined the phenomenon qualitatively in detail. These studies have provided insights into the characteristics of WMN, its functions, and its implications for human interaction, exploring its role in conversational repair (Myrendal, forthcoming), disagreements (Myrendal, 2019), and semantic conflicts about moral issues (Myrendal & Larsson, forthcoming). However, the fact that a corpus of WMNs has been lacking means that results so far have been mostly qualitative.

In addition to the theoretical contributions of the annotation schema and instructions, the resulting annotations facilitate future quantitative and qualitative investigation of WMN. These annotations can also be used to identify potentially controversial or problematic terms, and to investigate the mechanisms of conceptual or lexico-semantic alignment. Together, this foundational corpus and previous qualitative analyses contribute to both the theoretical understanding and practical applications of WMN, for example in developing automated dialogue systems capable of detecting, negotiating and adapting to unexpected, problematic or controversial word usages.

The paper is organized as follows: Section 2 reviews previous work on WMN and related concepts, and is followed by a detailed definition of WMN in Section 3. In Section 4, we outline the corpora that form the raw materials for the NeWMe corpus and explain our annotation procedure. Section 5 presents an analysis of our annotations, and Section 6 describes the inter-annotator agreement study. Finally, we conclude and discuss directions for future work in Section 7.

## 2. Related work

Repair and clarification are fundamental mechanisms for managing communication breakdowns, and they have been studied extensively in both Second Language Acquisition (SLA) and Conversation Analysis (CA). While SLA often focuses on repair in learning contexts, CA examines repair as a broader phenomenon in natural interaction.

Repair involves interrupting the conversational flow to resolve issues related to speaking, hearing, or understanding. A frequent feature of dialogue, repairs occur roughly once every 25 words in conversational speech (Hough & Purver, 2013). They play a crucial role in ensuring the smooth progression of interaction by maintaining or restoring intersubjectivity (Schegloff, 2007). Repairs can target a range of problems, from lexical ambiguities, such as the use of a problematic word, to more general issues like mishearing or disruptions in turn-taking. WMN sequences can be regarded as a type of conversational repair specifically targeting word meaning.

Clarification requests, a key form of other-initiated repair, play an important role in addressing breakdowns in understanding and managing interactional problems. As Purver (2004) highlights, clarification requests vary
widely in both their form and the type of information they seek to resolve. They may take explicit forms, such as
detailed queries (e.g., “What do you mean by X?”) or minimal and ambiguous expressions like “What?” that rely
on context and intonation for interpretation. These diverse forms reflect the flexibility of clarification requests in
targeting different aspects of an utterance, from the meaning of specific words to their appropriateness in context.

Purver’s taxonomy identifies explicit forms that clearly specify the issue and minimal forms that rely on context and intonation to clarify ambiguities. For instance, explicit queries like “What do you mean by X?” demand detailed responses, while minimal forms such as “X?” or “What?” signal a general difficulty with understanding the prior turn without pinpointing a specific trouble source. Drew (1997) explores these minimal repair initiators, referred to as “open-class repair initiators”, which do not locate a specific repairable but instead indicate a broader problem with the prior utterance. These initiators are particularly relevant when a speaker signals an issue in understanding the sequential connection of an utterance rather than a specific lexical ambiguity. Drew’s analysis highlights how open-class repair initiators are frequently used when speakers have difficulty grasping why something was said or how it relates to the prior discourse, underlining the connection between repair and alignment in conversation.

Research within the field of SLA has shown that during communication breakdowns, such as non-understanding of specific words, participants redirect their attention to the problematic words within the interaction. In such cases, participants shift their attention from conversational progress to addressing the meaning of specific words or phrases (Long, 1996). This negotiation typically follows a structured sequence: a **trigger**, which denotes the problematic word usage; an **indicator**, signaling the problem; and a **response**, aimed at resolving it, sometimes followed by a **reaction to the response** to conclude the sequence.

Indicators, the second component of this type of sequence, play a critical role in highlighting perceived problematic word usage. While they are often clarification requests, they can also take other forms, ranging from explicit meta-comments that directly ask for definitions to more implicit cues, such as silences or gestures signaling confusion (Pitzl, 2005; Vasseur et al. 1996). The nature and explicitness of the indicator often shape how the subsequent response will address the issue, determining whether the negotiation of meaning successfully resolves the misunderstanding.

Varonis and Gass (1985) conceptualize these breakdowns as vertical divergences in a horizontal conversational flow, with side-sequences resolving the issue before the conversation resumes. While this model was initially developed in SLA research, its applicability extends to natural, symmetrical conversations, as broader conversational repair patterns often align with this structure. Research related to non-understanding (Myrendal, forthcoming) similarly observes this pattern, where conversations shift vertically to address specific lexical misunderstandings before seamlessly returning to the main topic.

Larsson and Myrendal (2017a, 2017b) formalize semantic coordination by modeling WMNs through dialogue acts and meaning update functions. Their work synthesizes taxonomies of dialogue acts, distinguishing negotiation sequences based on comprehension difficulties versus disagreement over word choice. By linking these dialogue acts to specific types of semantic updates, they provide a framework for understanding how word meanings evolve dynamically in interaction.

Noble et al. (2021) build on these concepts by formalizing and annotating WMN as a structured interaction game with defined rules and semantic update functions. Their study uniquely focuses on WMNs in social media contexts, using a regular expression to identify potential sequences on Twitter, such as those containing “What do you mean by X?”. Annotators categorized WMNs as stemming from non-understanding or disagreement, aligning with prior models but emphasizing their utility in online discussions. Noble et al.’s work addresses the gap in prior research by providing a formal framework and annotation schema specifically for WMNs in online interaction, highlighting their role in semantic alignment and change.

Despite the insights from prior studies into conversational repair, clarification requests, and word meaning negotiation, no comprehensive corpus dedicated to WMN has been established until now. Similarly, no existing
annotation scheme exists that captures the nuanced distinction between types of WMNs, nor do they address the interplay between situated and potential meanings. Our study fills this gap by introducing the NeWMe Corpus (Negotiating Word Meaning), the first large-scale manually annotated corpus specifically designed for the study of WMN in diverse interactional settings. Unlike a traditional corpus that consists of newly collected linguistic data, the NeWMe Corpus does not contain original material but instead compiles instances of WMNs and related phenomena from existing corpora. In that way, it functions as a “virtual corpus” that provides standoff annotations and tools for accessing and analyzing WMN instances in a structured and convenient way. This resource lays the groundwork for future quantitative and qualitative research into semantic alignment and the dynamics of meaning negotiation. By building on and extending the theoretical frameworks outlined in this section, our contribution advances the empirical study of WMN, providing a gold standard for analyzing these interactional phenomena.

## 3. Word Meaning Negotiation

In this section, we introduce the established concept of Word Meaning Negotiation (WMN), drawing on previous research to provide a theoretical basis for our work. While this section does not present a novel contribution, it is essential for framing the subsequent presentation of our annotation scheme and analysis. WMN, as defined by Myrendal (2015, p 19), refers to “Instances in communication where participants explicitly negotiate between themselves their respective takes on the situated meaning of a particular word, and/or the meaning potential of that word.”

A core component of WMN is the *meta-linguistic shift*, formally defined as a conversational transition in which participants move from discussing the primary topic of interaction to explicitly addressing and negotiating the meaning, interpretation, or use of a specific word or phrase within the interaction. This shift aligns with Clark's (1996) notion of communication occurring on two parallel tracks. Track 1 deals with the “official business” of the conversation, focusing on the primary topic under discussion. Track 2, on the other hand, handles the meta-communicative aspects of interaction, ensuring the conversation runs smoothly by addressing coordination, turn-taking, and grounding. WMN operates within this second track, as it revolves around managing lexical or semantic issues that arise in the course of conversation. By engaging in WMN, participants collaboratively address and negotiate ambiguities or disagreements about word meanings, enabling the conversation on Track 1 to proceed effectively. This interplay between the two tracks highlights the dynamic nature of communication, where both the content and the process are managed simultaneously.

The minimal WMN sequence contains a three-turn exchange, Trigger-Indicator-Response (T-I-R), where the first and second turns are produced by two different people.

1. **Trigger Turn:** This is the initial use of the word that later becomes the focus of negotiation. The word might be a single word form or an expression (e.g., “kick the bucket”).

2. **Indicator Turn:** Following the trigger, this turn signals the need to discuss or clarify the word's meaning. It may come in the form of a direct request for clarification of meaning (a meta-linguistic clarification request) or as a challenge to the appropriateness or meaning of the word in the given context (a meta-linguistic objection).

3. **Response Turn(s):** This turn reflects a meta-linguistic shift, where the discussion moves from the initial topic to focus explicitly on the meaning of the word in question. The shift may not be completely separate from the original discussion topic, but clearly highlights the word's meaning as a central element of the exchange.

Importantly, WMNs do not always need to result in agreement about the word's meaning to be considered WMNs. Even if participants fail to fully understand or align on a shared interpretation of the meaning of the trigger word, the sequence still counts as a WMN as long as the conversational focus temporarily centers on negotiating the meaning of the word.

WMNs can be initiated in different ways, but often originate in either insufficient understanding of word meaning (this type of WMN is called a NON, short for “non-understanding WMN”) or disagreement about what a word
can or should mean in a specific situation (this type of WMN is called a DIN, short for “disagreement WMN”) (Myrendal, 2015).

## Example of WMN Caused by Non-Understanding of Word Meaning (NON)

S1: I'm going to the doctor to get a full body scan tomorrow.

S2: What do you mean by full body scan?

S1: I mean a kind of X-ray where they can see all of the inflamed parts.

This example, taken from Myrendal (2015), illustrates a scenario where S1's use of the term “full body scan” serves as the **trigger**, introducing a word which is not fully understandable to S2. S2 then produces an **indicator**, explicitly requesting clarification about the meaning of “full body scan,” making this phrase the trigger. In **response**, S1 provides an explanation, elaborating on the word to address the lack of understanding. This sequence demonstrates how WMNs initiated by non-understanding (NONs) focus on clarifying the meaning of specific terms to maintain mutual understanding in the conversation.

## Example of WMN Caused by Disagreement about Word Meaning (DIN)

S1: Telling children about Santa Claus is straight up lying to them.

S2: That's not what lying means at all!

S1: Of course it is, lying means not telling the truth and everyone knows Santa doesn't exist.

This example, drawn from Myrendal & Larsson (forthcoming), illustrates a WMN caused by disagreement about word meaning (DIN), where the focus shifts to negotiating differing perspectives on the meaning of a word. Here, S1's initial statement introduces the word “lying,” which serves as the **trigger**. S2 challenges this usage by providing an **indicator**, asserting that the term “lying” does not apply in the given context and objecting to its use. In **response**, S1 elaborates on their understanding of the word, reinforcing their interpretation and connecting it to the situation at hand.

Once a WMN has been started, the focus of the negotiation may center on either the semantic properties of the word itself - what Norén and Linell (2007) refer to as meaning potentials - or the specific details of the situated context in which the word is used. Meaning potentials, as defined by Norén and Linell (2007), encompass the set of abstract properties of a word that, when combined with contextual factors such as linguistic co-text and situational conditions, make possible all reasonably correct or contextually plausible interpretations of that word. These meaning potentials are not static dictionary definitions but dynamic properties shaped by the accumulated interactional experiences of both individuals and the broader language community over time. In contrast, situated meanings emerge in real-time interaction, where the meaning potential of a word is activated and shaped by the immediate context. While meaning potentials provide the flexible foundation of a word's possible uses, situated meanings are collaboratively constructed and negotiated in situ by conversational participants.

WMNs can focus on these two main aspects of meaning: **potential meaning** and **situated meaning**. When the focus is on potential meaning, participants discuss what a word can mean in a broader sense, beyond the immediate context. This might occur in NONs if S2 does not know the word at all or is unfamiliar with the sense in which it is being used (even if it is a well-established sense). In DINs, potential meaning becomes the focus if S2 challenges S1's usage, arguing that it does not align with any established meaning of the word.

On the other hand, WMNs may center on situated meaning, where the negotiation revolves around what the word means in the specific situation being discussed, with less emphasis on its broader usage. In NONs, this could involve S2 struggling to understand how the word applies to the particular object or scenario at hand. In DINs, situated meaning is at the core when S2 points out that S1's interpretation does not fit the specifics of the current situation.
WMNs can also simultaneously center on **both** potential meanings and situated meanings. In such cases, the negotiation explicitly addresses both what a word can mean in a broader, general sense and what it means in the specific context of the conversation. For instance, S2 might confront S1 by referring to a general definition of the word (e.g., from a dictionary), emphasizing its potential meaning, while S1 explains their intended meaning within the specific situation, focusing on its situated use. Alternatively, S2 might argue that S1's use of the word is unconventional in the given context (situated meaning), prompting S1 to defend their usage by appealing to its broader, standard interpretation (potential meaning). While situated aspects of meaning are always inherently present - since conversations always occur within specific contexts - we consider that WMNs explicitly focus on both dimensions when the negotiation clearly addresses both what the word means in the immediate situation and what it can mean beyond that context.

# 4. Corpus Construction

In this section, we outline the methodology used to create the corpus. Following the selection of a diverse sample of corpora for annotation, we employed a regular expression-based approach to pre-select utterances likely to contain WMN indicators (Section 4.1). Section 4.2 introduces the annotation labels and provides illustrative examples for each, highlighting their application in capturing different types of WMNs and related phenomena. We also outline the annotation interface and process, which was conducted by two authors of this paper, detailing the creation of comprehensive annotation guidelines and the calculation of inter-annotator agreement (IAA) to ensure reliability.

## 4.1 Data selection

### 4.1.1 Corpora used

The NeWMe corpus consists of a collection of annotated WMNs and related phenomena from existing corpora. In selecting the corpora to be included in the NeWMe Corpus, we prioritized sources that offer a range of interactional settings and conversational structures, enabling analyses of WMN across varied contexts. Our aim was to include both spoken and written interactions, spanning dyadic spoken conversations, multiparty spoken interactions, and written online multiparty discussions. This variety ensures a broad representation of how WMNs manifest across different interactional settings, including debates, informal conversations, and other diverse topics and situations. We selected three corpora for annotation:

* The **Switchboard Dialog Act Corpus** (Stolcke et al., 2000) contains 1 155 dyadic phone conversations. Dialogue participants, all from the United States, did not know each other, and were assigned a topic to discuss (e.g., movies, food, vacation spots, recycling, etc.). We access the corpus through the ConvoKit Python library (Chang et al., 2020).

* The **British National Corpus (BNC)** (BNC Consortium, 2007). We use a portion of the spoken part of the BNC, corresponding to 730 spoken interactions. The data consists of transcripts of spoken conversations recorded in different conversational settings, including lectures, meetings, interviews, medical consultations and news broadcasts, among others.

* The **Winning Arguments (ChangeMyView) Corpus** (Tan et al., 2016), hereafter referred to as **Reddit**, contains conversations from the r/ChangeMyView一条subreddit posted between 2013 and 2015. In this一条subreddit, the original poster of a thread states their view on some topic and challenges other users to try to change their opinion, which often generates debate-like conversations with users exposing their arguments. We chose this corpus because we expect lexico-semantic alignment to be particularly important in such a setting: users need to make sure that they are debating about the same things, and disagreements are likely to surface. Reddit contains 3 051 conversations and is also available as part of the ConvoKit package. A specificity of Reddit is its hierarchical structure, with parallel subthreads. We treat each post as an utterance.
### 4.1.2 Candidate selection with regular expressions

As discussed in Section 2, previous research used regular expression matching to identify potential WMNs in Twitter data (Noble et al., 2021). This approach focused on the indicator turn, which typically exhibits less variation compared to the trigger and negotiation phases of a WMN. However, the regular expression employed by Noble et al. was limited to simple variations of the question “What do you mean by X?”, such as “what do you actually mean by.” In this study, we developed a more extensive list of regular expressions, hypothesizing that they could better capture the diversity of indicator turns.

As a first step towards choosing relevant expressions, we took advantage of the dialogue act annotations in the Switchboard corpus (Stolcke et al., 2000). We compiled all utterances which were tagged with a dialogue act label involving a question (excluding rhetorical questions) or a signal of non-understanding.<fnref n="6" /> Through manual inspection of the first 3 000 utterances tagged with these labels, we identified various expressions such as “what is,” “what is the difference between,” and repetitions of words and phrases from the immediately preceding turn, followed by a question mark (e.g., S1: “I do aerobics, step classes, and…” S2: “step classes?”), which we refer to as the “repetition pattern.”<fnref n="7" /> We supplemented this list with expressions based on our own intuitions (e.g., “can you define”) and insights from prior research (e.g., “this is not X”) (Myrendal, 2015).

We used a total of 30 regular expressions alongside the repetition pattern. The complete list of simplified expressions is presented in Section 6.2. It includes variations of expressions such as “what X means”, “in what way/sense?”, “the meaning of”, and others.

Certain adaptations were necessary to take into account the formatting specificities of the included corpora and ensure that the regular expressions would match all relevant candidates. Specifically, several regular expressions were adjusted for Switchboard by making question marks optional, since we observed that they are sometimes omitted (see Example 1 in Section 4.2.1). Additionally, before checking for matches on Switchboard, we simplified utterances by temporarily removing indications of laughter, information in curly brackets, and transcription conventions related to spoken phenomena, such as symbols like “+” or “--”. For simplicity, we chose to normalize the transcriptions rather than adjusting the regular expressions, which is more error-prone. For the Reddit data, the regular expression search excluded all cited passages (i.e., where a user quotes text from a previous message), as any relevant matches were already captured in the original posts.

We examined every utterance<fnref n="8" /> of the three aforementioned corpora for regular expression matches. Any utterance that matched at least one regular expression was retained as an annotation instance. This procedure obtained 1 305 instances from Switchboard, 2 976 from Reddit and 4 032 from the BNC.

## 4.2 Annotation

### 4.2.1 Labels and spans

This section presents the final set of annotation labels used in this study. The annotation process began with identifying the type of phenomenon present, assigning a corresponding phenomenon label. Based on this classification, relevant portions of text were marked with span labels, each representing a distinct component of a WMN. When an instance was annotated as a WMN, we additionally annotated the kind of meaning that was being discussed. More information and details can be found in the annotation guidelines, which will be provided alongside the annotations.

<footnote n="6">⁶ We used the tags *qy*, *qw*, *qy^d*, *qo*, *br*, *qw^d* and ^g (http://compprag.christopherpotts.net/swda.html) (accessed January 29, 2025).</footnote>

<footnote n="7">⁷ For this pattern, we excluded repetitions consisting of “yes ?”, “yeah ?” and “no ?”, as we observed they had a very large number of matches, especially in the BNC, but were never indicative of a WMN.</footnote>

<footnote n="8">⁸ We excluded the first utterance (the title, in Reddit), as the regular expressions should help find potential indicators, which, in a WMN, must happen in a turn posterior to the trigger turn.</footnote>
## Types of complete WMNs: NON, DIN and Other

As introduced in Section 3, we distinguish between two main types of: WMN: **NONs** (non-understanding WMNs), driven by insufficient understanding that prompts S2 to request clarification; and **DINs** (disagreement WMNs), where S2 challenges or objects to S1's use of a word, focusing the negotiation on resolving the disagreement. During annotation, we also identified a third category, WMN: **OTHER**, for cases where word meaning was discussed without non-understanding or disagreement. These typically involve situations where S2 asks about a word and/or suggests an alternative word, which S1 then confirms as appropriate.

Examples of each type of WMN are shown in Examples 1, 2 and 3.<fnref n="9" />

<figure><img src="image_4.png" /></figure>

### Example 1 NON from Switchboard about the word “liberal”

<figure><img src="image_5.png" /></figure>

### Example 2 DIN from Reddit about the word “waffle.”

<figure><img src="image_6.png" /></figure>

### Example 3 WMN: OTHER from the BNC about the word “vulnerable”

For WMNs, we also annotated the type of meaning. As described in Section 3, this could be potential meaning, situated meaning, or both. Example 1 above focuses on potential meaning, Example 2 discusses both kinds of meaning, and Example 3 addresses situated meaning.

## Incomplete WMNs and Distractors

<footnote n="9">⁹ We use "(...)" to indicate omitted portions of the conversation, including partial or full turns, that are not relevant to the example for the sake of brevity.</footnote>
While WMNs involve specific sequences where participants explicitly negotiate word meanings, related phenomena may only partially meet these criteria or exhibit distinct but similar interactional patterns. During annotation, we identified such cases as either *incomplete WMNs* (SELF-INITIATED MEANING NEGOTIATION, NON-PURSUED, WITHOUT TRIGGER) or *distractors* (REFERENCE/NE, OTHER KINDS OF CLARIFICATION REQUESTS), serving as negative examples for distinguishing true WMNs. Below, we outline these categories and their defining characteristics and show examples from the corpus:

SELF-INITIATED MEANING NEGOTIATION (SIMN) occurs when the same person produces both the trigger word and the indicator. For example, the speaker may ask someone else to clarify the meaning of a word just used, to see if that word was understood.

S1: Not a good idea. Erm have you heard of half life ?
S2: Yeah.
S1: What does that mean ?
S2: that, that's how long it takes for half of the radioactive isotopes to disappear.
S1: Great, yeah, that 's a wonderful definition.

**Example 4** A SIMN from the BNC about “half life”

NON-PURSUED sequences occur when an attempt to initiate a WMN is not successful, i.e., when there is no uptake on the indicator. These cases involve a trigger and an indicator but lack a response turn addressing the semantic issue (i.e., there is no meta-linguistic shift). There is an attempt to initiate a negotiation targeting word meaning, but it does not materialize.

S1: (...) I am telling you race is not a scientific description and its use as a popular culture description causes hate and ignorance. (...)
S2: (...) Do you know what you mean by "unscientific", or is it just a non-declarative placeholder speech act you use when you have attitudes of aversion? (...)
S1: Wow you are putting allot of effort into avoiding my question. I was careful to put it into the most basic terms and yet there was still a communication failure, no problem, lets break out the crayola's. Do you hold the position that race is a scientific descriptor? Your answer should be framed as a "yes", "no" or "unknown". Seriously, dont over think this one, just a one word answer. You know what, let me pull it out of this paragraph and restate, again, no tricks, just a simple one word answer is all that required. Also, if you are worried about tricks go ahead and ask any questions you need, look up any references for clarification, just dont let it interfere with your response. 

\n\nDo you hold the position that race is an accepted scientific descriptor? Your answer should be framed as a "yes", "no" or "unknown".

**Example 5** A NON-PURSUED sequence from Reddit about the word “(un)scientific.” In this case, the utterance containing the indicator receives an answer by S1, but the answer does not address the indicator.

In sequences categorized as WITHOUT TRIGGER, the problematic word or phrase is introduced directly in the indicator sentence, without any prior use of the word in the conversation. As a result, the initial trigger component of the three-part exchange required for a sequence to qualify as a WMN (T-I-R) is missing. This type of negotiation may occur when the need to clarify a word's meaning emerges from external factors or prior knowledge (e.g., an external resource, a speaker's experience, or the conversational context) rather than from the immediate dialogue. It can also arise when the trigger word is not explicitly used but the conversation addresses a related topic (e.g., S1 discusses feminism without mentioning the word, prompting S2 to indicate a different understanding of “feminism”).

S1: (...) what do you think about this new menu for the canteen at Digby's Ballbearings ?
S2: Crunchy nut salad, t , what 's tortellini ?

S1: Pasta, stuffed with spinach and cheese (..)

**Example 6** A WITHOUT TRIGGER sequence from the BNC

Distractors involve sequences that resemble WMNs but do not meet the criteria because they do not contain
negotiation of word meaning. REFERENCE/NAMED ENTITY (NE) sequences focus on resolving referential ambiguity
rather than negotiating the meaning of a word (Example 7). Sequences identified as OTHER KINDS OF
CLARIFICATION REQUESTS involve clarification of broader semantic content rather than a specific word's meaning.
Often, the request targets the entire utterance rather than an individual lexical item (Example 8).

S1: they'd have their smog alerts where you'd have to stay indoors for so many hours with an air conditioner. And, of course, they don't have **that** here in Texas. So, there's...

S2: You mean they don't have the, uh, the smog alerts?

**Example 7** A REFERENCE/NAMED ENTITY sequence from Switchboard

S1: For this to change my view I'd need strong evidence that presumably lower interest rates on lons from banks have lead to a substantial increase in quality of life as they have increased.

S2: I'm not sure what you mean here. Lower interest rates on a loan mean that your monthly payments are lower, and it's easier to pay off / you have more money available for other things. That's a quality of life increase.

Example 8 An OTHER KIND OF CLARIFICATION REQUEST from Reddit

Other labels: NOTHING and IMPOSSIBLE TO ANNOTATE

When none of the above labels applied, an instance was labeled as NOTHING. If the annotator suspected that a
phenomenon was present but could not be sure due to transcription errors or incompleteness, the instance was
labeled as IMPOSSIBLE TO ANNOTATE: In the example below from the BNC, the turn that may have contained the
trigger could not be properly/fully transcribed:

S1: [UNCLEAR] was n't [UNCLEAR]

S2: Yes, that 's quite nice . So it 's dreamlike what do you mean by that . What 's he doing then ?

S1: Well it 's not very clear whether he 's actually thinking or seeing it.

**Example 9** An instance from the BNC labeled as IMPOSSIBLE TO ANNOTATE

Spans: trigger, indicator and negotiation

In our annotation process, a span refers to a specific segment of an utterance that is chosen and labeled with one
of three categories: Trigger, Indicator, or Negotiation. When the annotator determined that there was a (complete
or incomplete) WMN sequence or a distractor, the spans corresponding to the three categories were annotated.
The **trigger** span was used to tag all problematic word usages, and was annotated for all phenomena except **OTHER**
**KINDS OF CLARIFICATION REQUESTS**, where the request does not target a word or phrase. The **indicator** span was
used to highlight the utterance(s) indicating the problem and was used for all phenomena. Finally, the **negotiation**
span was used to demarcate the response turns addressing a problem with word meaning, if present. See the
passages highlighted in Example 3 for illustration.
### 4.2.2 Annotation interface

We used the LabelStudio<fnref n="10" /> platform to perform annotation. An instance of annotation consists of a full conversation where one utterance has been matched by one or more regular expressions. The strings matched by the regular expressions were highlighted as potential indicators. The annotator needed to determine whether an instance contained a WMN or some related phenomenon, and, if necessary, select the spans corresponding to its different parts. The annotation focused around the utterance containing the potential indicator or in its close vicinity.¹¹ A screen capture of the annotation interface is presented in the Appendix (Figure 1).

### 4.2.3 Annotation procedure

One author of this paper annotated around 81% of the BNC instances and another author annotated the Reddit and Switchboard data as well as the remaining portion of the BNC data. The annotators worked individually but held regular meetings to discuss progress as well as complicated or unexpected cases. These meetings helped not only to ensure consistency in annotation, but also to refine and adapt the set of labels based on the observed data. The decisions taken during these meetings were documented in the annotation guidelines, which were written and updated concurrently.

Once the first iteration of annotation was complete, there was a second round where the two authors went through all instances that had been labeled with some phenomenon and ensured that they were consistent with the final set of labels and their definitions as described in the guidelines. At the end of this process, a sample of 254 instances (175 and 79 by each annotator) was randomly compiled covering multiple cases concerning each label. These instances were manually reviewed by the other author to assess agreement. In cases of disagreement, the two authors discussed the annotations in an effort to reach consensus. However, in a small number of instances, consensus could not be achieved, and the original annotator's choice was kept. Most disagreements stemmed from differing interpretations of the specific conversational context. WMN annotation is an inherently subjective task and the presence of some disagreement is expected. Some form of disagreement remained only in 14 instances of the sample, 10 of which concerned the label used, and 4 others about the type of meaning, resulting in an agreement of 94-96% between the two authors, compared to a 86-90% agreement before discussion.

The speed of annotation varied depending on the corpus, how many interesting cases were found, and the acquired experience both with the annotation task and the LabelStudio platform. Uninteresting cases could often be quickly detected and annotated as NOTHING. Reddit data had the additional complexity of having to keep track of the response subthreads. The average annotation speed was estimated to be 1.4 instances per minute for one author and 1.05 instances for the other.

## 5. Analysis

### 5.1 Corpus statistics

Table 1 presents the number of phenomena found in the WMN corpus. We annotated a total of 404 WMNs: 215 NONs, 169 DINs and 20 WMN: OTHER. WMN types are not evenly distributed across sources: A large majority of DINs (94%) are found in Reddit data. This aligns with findings that anonymous online platforms often foster interest-based selectivity and reduced multidimensionality in social affiliations, which can amplify disagreement (Barnidge, 2017). Unlike social media networks that articulate connections across diverse social dimensions (Boyd & Ellison, 2007; Ellison et al., 2007), anonymous forums lack shared common ground and non-verbal cues, making disagreement more frequent and fostering environments conducive to word meaning negotiation. In

<footnote n="10">¹⁰ https://labelstud.io/

¹¹ We also considered phenomena taking place in other parts of the conversation if they were detected by the annotators while reading the context to understand the situation.</footnote>
contrast, most NONs (69%) are found in spoken conversation. This could also be expected, as the asynchronicity of online forums gives participants time to reflect upon or look up word meanings (Myrendal, 2019); while the immediacy of spoken communication requires solving the understanding problems on the spot in order to successfully continue communication. With only 20 instances in the whole corpus, WMN: OTHER was not very common, but not an isolated case either.

<table><caption>**Table 1** Frequency of every phenomenon in the WMN corpus by source</caption><thead><tr><th></th><th>BNC</th><th>Reddit</th><th>Switchboard</th><th>Total</th></tr></thead><tbody><tr><td>NONs</td><td>116</td><td>66</td><td>33</td><td>215</td></tr><tr><td>DINs</td><td>11</td><td>158</td><td>0</td><td>169</td></tr><tr><td>WMN: Other</td><td>14</td><td>3</td><td>3</td><td>20</td></tr><tr><td>Non-pursued</td><td>4</td><td>197</td><td>2</td><td>203</td></tr><tr><td>SIMN</td><td>37</td><td>2</td><td>3</td><td>42</td></tr><tr><td>Without trigger</td><td>10</td><td>0</td><td>2</td><td>12</td></tr><tr><td>Reference/NE</td><td>7</td><td>3</td><td>18</td><td>28</td></tr><tr><td>Other kinds of clarification requests</td><td>15</td><td>109</td><td>49</td><td>173</td></tr><tr><td>Nothing</td><td>3746</td><td>2353</td><td>1188</td><td>7287</td></tr><tr><td>Impossible to annotate</td><td>24</td><td>1</td><td>0</td><td>25</td></tr><tr><td>Total</td><td>3984</td><td>2892</td><td>1298</td><td>8174</td></tr><tr><td>Total phenomena</td><td>214</td><td>538</td><td>110</td><td>721</td></tr></tbody></table>

The distribution of incomplete WMNs also paints an interesting picture. NON-PURSUED sequences are almost exclusively (97%) found in Reddit data, and they are more numerous than DINs in this subset of the data (197 vs 158). This makes sense, as it is much easier to stop a thread, ignore a post or change the topic in online forums than it is to ignore a clarification request in real-time spoken conversation. In spoken interaction, research within Conversation Analysis (CA) has established a strong preference for an answer to come in the next turn when a direct question is asked (Sacks et al., 1974; Heritage, 1984). Questions are typically treated as making a response conditionally relevant, and failing to provide an answer is often interpreted as a breach of normative expectations. This orientation underscores the importance of maintaining interactional continuity and progressivity in spoken interaction (Stivers & Robinson, 2006; Schegloff, 2007). Furthermore, Karafoti (2021) shows that delayed or missing responses stand out and often require extra effort, like rephrasing or explaining, to repair. In contrast, asynchronous online forums like Reddit lack the real-time pressure of spoken interaction, allowing participants to ignore posts or disengage from threads without facing social consequences. This flexibility leads to a higher number of NON-PURSUED WMNs in Reddit data, where many posts (70%) remain unanswered or receive responses that don't engage in word meaning negotiation (30%). Unlike face-to-face communication, where conversations rely on immediate and accountable responses (Schegloff, 2007; Pomerantz & Heritage, 2013; Karafoti, 2021), online interaction reduces the obligation to reply, making non-responses less noticeable and less socially significant.

In contrast to the NON-PURSUED sequences, SELF-INITIATED MEANING NEGOTIATION (SIMNs) were most commonly found in the BNC data. These cases often involve classroom-like interactions, where a teacher introduces a word and asks students if they know its meaning, prompting a meta-linguistic discussion for
educational purposes. Instances WITHOUT TRIGGER were not very common (12 in total). These instances typically arise when the problematic word comes from a source external to the conversation (e.g., someone is reading something and finds a word they don't understand), as well as in teacher-student interactions. With 173 instances, OTHER KINDS OF CLARIFICATION REQUESTS were among the most commonly encountered phenomena, especially in the Reddit data. In contrast, the other distractor, REFERENCE/NE was less frequent, appearing only 28 times.

We also want to draw attention to the large number of irrelevant cases, annotated as NOTHING, which provides insight into the effectiveness of the regular expressions used to capture relevant phenomena.<fnref n="12" /> Only about 5% and 8% (respectively) of the retrieved BNC and Switchboard instances contained interesting phenomena, compared to 19% of Reddit. However, this drops to 8% for Reddit when considering only complete WMN sequences. While NOTHING instances are often quick to annotate, these percentages point to a clear opportunity for improvement. Future work could focus on refining the regular expressions or using semi-automatic methods to optimize and speed up the annotation process, pre-selecting instances that are more likely to contain a relevant phenomenon (see Section 5.2 for more details).

To provide a clearer picture of the prevalence of identified WMNs in our data, we note that regular expressions returned matches in 1 863 distinct conversations (38% of all available conversations), but only 276 of these contained complete WMNs (6% of all available conversations). Looking at these results by corpus, BNC was the most dense in WMN: 13% of all BNC conversations contained at least one WMN, compared to a 3% and 5% in Switchboard and Reddit, respectively. It is important to note, however, that our analysis is limited to instances matched by our regular expressions, and we do not have an estimate of how many WMNs may have been missed.

<table><caption>Table 2</caption><thead><tr><th></th><th></th><th>BNC</th><th>Reddit</th><th>Switchboard</th><th>Total</th></tr></thead><tbody><tr><td rowspan="3">NON</td><td>situated</td><td>88 (75.9%)</td><td>52 (78.8%)</td><td>17 (51.5%)</td><td>157 (73.0%)</td></tr><tr><td>potential</td><td>20 (17.2%)</td><td>8 (12.1%)</td><td>13 (39.4%)</td><td>41 (19.1%)</td></tr><tr><td>both</td><td>8 (6.9%)</td><td>6 (9.1%)</td><td>3 (9.1%)</td><td>17 (7.9%)</td></tr><tr><td rowspan="3">DIN</td><td>situated</td><td>5 (45.5%)</td><td>32 (20.3%)</td><td>0</td><td>37 (21.9%)</td></tr><tr><td>potential</td><td>0</td><td>58 (36.7%)</td><td>0</td><td>58 (34.3%)</td></tr><tr><td>both</td><td>6 (54.5%)</td><td>68 (43.0%)</td><td>0</td><td>74 (43.8%)</td></tr><tr><td rowspan="3">Other</td><td>situated</td><td>14 (100%)</td><td>2 (66.7%)</td><td>3 (100%)</td><td>19 (95.0%)</td></tr><tr><td>potential</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>both</td><td>0</td><td>1 (33.0%)</td><td>0</td><td>1 (5.0%)</td></tr><tr><td rowspan="3">All WMNs</td><td>situated</td><td>107 (75.9%)</td><td>86 (37.9%)</td><td>20 (55.6%)</td><td>213 (52.7%)</td></tr><tr><td>potential</td><td>20 (14.2%)</td><td>66 (29.1%)</td><td>13 (36.1%)</td><td>99 (24.5%)</td></tr><tr><td>both</td><td>14 (9.9%)</td><td>75 (33.0%)</td><td>3 (8.3%)</td><td>92 (22.8%)</td></tr></tbody></table>

### Frequency of the type of meaning by WMN type and corpus source

In Table 2, we present statistics on the type of meaning of each WMN, categorized by source. When examining patterns for each type of WMN, we observe that NONs and WMN: OTHER most frequently involve discussions about the *situated meaning* of the word. This indicates that insufficient understanding, in our corpus, is mostly tied to the specific way a word is used or its application to a specific situation. WMN: OTHER interactions often involve corrections or suggestions of alternative lexical choices or phrasing, and thus their tendency to relate to

<footnote n="12">¹² We exclude from the corpus statistics any instances where a phenomenon was present but had already been annotated in another instance from the same conversation.</footnote>
situated meaning is not surprising. DINs, however, show a different pattern: these interactions more often address the *potential meaning* of a word, or a combination of both its situated and potential meanings. In many cases, disagreement arises when participants bring up definitions of words as a way to defend their viewpoints, highlighting the potential meaning of the word. See for instance Example 10 below. Sometimes, post authors use definitions to contrast them with the situated meaning proposed by another participant, which they may perceive as incorrect or inadequate.

S1: (...) you're saying that patriotism and similar impulses necessarily lead to hatred and discrimination. I think this is untrue, and that there are many people that are both patriotic and deeply empathetic toward people from different countries (...)

S2: Well what is patriotism then ?!

S1: Love of country. Love of one thing doesn't imply or lead to hatred of another. Nor does it require that you be vocal about it.

S2: Ok. I'm from eastern europe, so my view is clearly biased here, because the concept of "love of country" seems unnatural to me.

S1: Fair enough, but that's the [dictionary definition] (http://www.merriam-webster.com/dictionary/patriotism) of patriotism. Thinking it's weird is different than thinking it leads to negative feelings.

**Example 10** A DIN from Reddit discussing the potential meaning of the word “patriotism”

We also examine the characteristics of the annotated spans. For triggers, the majority (71%) consist of individual words, while 27% are multi-word phrases, and 2% involve a combination of both (e.g., “feminism” and “feminist movement”). Interestingly, the spoken corpora contain a higher proportion of phrase triggers, with 36% in the BNC and 39% in Switchboard, compared to 21% in Reddit. An interesting direction for future research would be to explore how the nature of the trigger —whether it is a single word or a (lexicalized) phrase— relates to the type of meaning being negotiated. Our data suggests that WMNs focusing on situated meaning are more likely to involve trigger phrases than those focusing on potential meaning. This pattern may arise because phrases inherently provide more contextual information, anchoring their meaning more directly to the specific utterance in which they appear. For example, in “What do you mean by ‘light meal’?”, the phrase “light meal” invites a discussion about the meaning of “meal” in this particular context (e.g., portion size or calorie content), reinforcing its situatedness. In contrast, individual words, such as “calculate” in “What do you mean by ‘calculate’?”, are more likely to invite discussion of their broader, more general meaning. Notably, phrases do not typically have potential meanings unless they are lexicalized expressions. Regarding the extent of negotiation, DINs tend to involve longer exchanges than NONs, averaging 7.2 turns compared to 3.5 turns. This difference aligns with expectations as well: NONs typically conclude once meaning is clarified, while in DINs each participant provides their own perspective and engages in debating word meaning. This pattern is consistent with Myrendal’s (2015) original study of WMN, which also found that NONs are typically brief and uniform, concluding within three to four turns on average. In contrast, DINs display much greater variability in length, in Myrendal’s study ranging from as few as five turns to as many as 85. In our corpus, the longest NON contains 27 turns in total, while the longest DIN spans 268 turns. This highlights the more elaborate and prolonged nature of DINs, where participants engage in extended exchanges to explore and debate different interpretations of word meaning.

## 5.2 Evaluating the success rate of regular expressions

In this section, we evaluate the effectiveness of our regular expression approach in identifying relevant phenomena and analyze variations in indicator expressions across the three corpora for NONs and DINs. By analyzing the success rates and outcomes of each pattern, we aim to understand their contributions to the annotation process and identify areas for improvement in future methodology refinements.

Table 3 displays the regular expressions (regexes) used to search for potential WMNs, alongside their performance metrics. The regexes are sorted from highest to lowest with regards to how many WMNs they produced in the searches. Each row corresponds to a specific regex, providing the following details:
* **Regex Name:** The name given to the regular expressions used to identify potential WMNs in the dataset.

* **Total Matches:** The number of instances in the dataset that matched the regex.

* **# WMNs:** The count of instances identified as WMNs among the matches.

* **% WMNs:** The proportion of matches that were confirmed as WMNs, reflecting the precision of the regex for WMN identification.

* **# Successful Matches:** Successful matches include both WMNs and other related discursive phenomena of interest to this study. Aside from WMNs, this also includes incomplete WMNs and distractors.

* **% Successful Matches:** The proportion of matches considered relevant to the broader analysis, including confirmed WMNs, incomplete WMNs, and distractors. This metric reflects the regex's overall effectiveness in capturing both strict WMNs and related phenomena of interest.

This table allows for a detailed comparison of the performance of each regex, displaying differences in their specificity and success rates for identifying WMNs or other phenomena considered relevant in this study. Regexes with higher percentages of successful matches (WMN or not) demonstrate greater precision, while those with high total matches but lower success rates highlight the trade-off between broad coverage and targeted accuracy.<fnref n="13" />,<fnref n="14" />

<table><thead><tr><th>Regex Name</th><th>Total matches</th><th># WMNs</th><th>% WMNs</th><th># Successful matches</th><th>% Successful matches</th></tr></thead><tbody><tr><td>you mean</td><td>834</td><td>172</td><td>20.6%</td><td>385</td><td>46.2%</td></tr><tr><td>can you define</td><td>853</td><td>149</td><td>17.5%</td><td>224</td><td>26.3%</td></tr><tr><td>definition of</td><td>481</td><td>121</td><td>25.2%</td><td>192</td><td>39.9%</td></tr><tr><td>what is</td><td>1484</td><td>65</td><td>4.9%</td><td>122</td><td>8.2%</td></tr><tr><td>what do you mean by</td><td>154</td><td>63</td><td>40.9%</td><td>114</td><td>74.0%</td></tr><tr><td>this is not X</td><td>1929</td><td>59</td><td>3.1%</td><td>118</td><td>6.1%</td></tr><tr><td>is that a/the</td><td>952</td><td>37</td><td>3.9%</td><td>80</td><td>8.4%</td></tr><tr><td>this isn't X</td><td>873</td><td>36</td><td>4.1%</td><td>61</td><td>7.00%</td></tr><tr><td>[repetition pattern]</td><td>1006</td><td>30</td><td>3.0%</td><td>34</td><td>3.4%</td></tr><tr><td>what X means</td><td>346</td><td>28</td><td>8.1%</td><td>47</td><td>13.6%</td></tr><tr><td>the term</td><td>157</td><td>20</td><td>12.7%</td><td>28</td><td>17.8%</td></tr><tr><td>in what way/sense</td><td>73</td><td>11</td><td>15.1%</td><td>19</td><td>26.0%</td></tr><tr><td>what does X mean</td><td>108</td><td>11</td><td>10.2%</td><td>33</td><td>30.6%</td></tr><tr><td>are you talking about</td><td>52</td><td>7</td><td>13.5%</td><td>20</td><td>38.5%</td></tr><tr><td>what is the difference between</td><td>70</td><td>7</td><td>10.0%</td><td>13</td><td>18.6%</td></tr><tr><td>isn't this...?</td><td>78</td><td>7</td><td>9.0%</td><td>10</td><td>12.8%</td></tr><tr><td>what sort/kind of</td><td>132</td><td>6</td><td>4.5%</td><td>8</td><td>6.1%</td></tr></tbody></table>

<footnote n="13">¹³ The regular expression “is that a/the” was excluded from the BNC corpus to speed up the annotation process. Early in the reference annotation phase, it became evident that while this expression generated a large number of matches, its success rate was very low. The cases listed in Table 3 for this regular expression in the BNC reflect instances where other regular expressions also matched the same utterance.</footnote>

<footnote n="14">¹⁴ Phenomena that were matched by more than one regular expression count as successfully identified instances for each of the matching regular expressions. For this reason, the numbers in this table do not align exactly with the global corpus statistics.</footnote>
<table><caption>**Table 3** Overall success of every regular expression</caption><tr><td>elaborate</td><td>119</td><td>6</td><td>5.0%</td><td>18</td><td>15.1%</td></tr><tr><td>the meaning of</td><td>51</td><td>5</td><td>9.8%</td><td>8</td><td>15.7%</td></tr><tr><td>is this not...?</td><td>35</td><td>4</td><td>11.4%</td><td>5</td><td>14.3%</td></tr><tr><td>are we talking</td><td>38</td><td>3</td><td>7.9%</td><td>9</td><td>23.7%</td></tr><tr><td>do you know what X is</td><td>43</td><td>2</td><td>4.7%</td><td>10</td><td>23.3%</td></tr><tr><td>is this what you mean</td><td>17</td><td>2</td><td>11.8%</td><td>8</td><td>47.1%</td></tr><tr><td>word X means</td><td>28</td><td>2</td><td>7.1%</td><td>3</td><td>10.7%</td></tr><tr><td>have you heard of</td><td>40</td><td>1</td><td>2.5%</td><td>3</td><td>7.5%</td></tr><tr><td>is that what X is</td><td>15</td><td>1</td><td>6.7%</td><td>2</td><td>13.3%</td></tr><tr><td>does X count as Y</td><td>6</td><td>1</td><td>16.7%</td><td>1</td><td>16.7%</td></tr><tr><td>how do you mean that</td><td>2</td><td>0</td><td>0.0%</td><td>2</td><td>100.0%</td></tr><tr><td>that would mean</td><td>48</td><td>0</td><td>0.0%</td><td>0</td><td>0.0%</td></tr><tr><td>is that what that/this is</td><td>1</td><td>0</td><td>0.0%</td><td>0</td><td>0.0%</td></tr><tr><td>depends on what you mean</td><td>2</td><td>0</td><td>0.0%</td><td>0</td><td>0.0%</td></tr></table>

When examining the table of regex patterns and their success rates in identifying instances of WMN, several observations stand out. First, the regex pattern “you mean” generates the highest number of successful matches of the tested expressions (172), indicating its common usage in interactions where participants initiate WMN by seeking clarification or questioning the meaning of words and phrases. In comparison, the regex “what do you mean by” - which subsumes the broader regex “you mean” - produces fewer successful matches (63) but has a higher precision than the broader regex (40.9% successful matches, compared to 20.6% for “you mean”). The pattern “you mean” is more open-ended and can appear in various contexts that may not always involve a direct question about word meaning, which could lead to a higher number of hits but a lower precision in identifying WMNs. In contrast, “What do you mean by”, which has a high precision in identifying WMNs, explicitly starts with an interrogative word followed by a direct address “you”, making it more likely to be used in contexts where semantic clarification is sought regarding a word’s meaning.

Further down the list, we find some regexes that produce a considerable amount of successful WMNs, but that have a very low precision rate. For example, the regex “what is”, “this is not X”, “this isn’t X”, “is that a/the” and the repetition pattern produce between 37 and 65 WMNs, but generate a very high amount of NOTHING. For instance, the regex “what is” generates 65 WMNs, which is significant. However, it also results in 1484 total hits, indicating a substantial number of irrelevant sequences that researchers must sort through to find the interesting cases. This highlights a common challenge with broader patterns: while they can produce a wide range of potential negotiations, their lower precision requires significant validation which can take a lot of time.

For brevity and ease of presentation, subsequent tables will include only the most productive regular expressions. Full versions of the tables can be found in the Appendix.

When looking at the success rate of each regex in relation to the three different corpora in Table 4, some similarities and differences appear. For example, the regex “you mean” displays varying levels of success across the corpora. In the BNC, it produced 424 matches, with 61 confirmed WMNs, resulting in a precision of 14.4%. In Switchboard, the regex generated 91 matches but achieved a higher precision of 20.9%, with 19 confirmed WMNs. In contrast, this regex performed much better in the Reddit corpus, producing 319 matches with 92 confirmed WMNs, achieving a precision of 28.8%. This strong performance in online argumentative settings
highlights the regex's relevance in text-based debates, where participants frequently engage in clarification or semantic negotiation, making it an effective tool for WMN detection in such settings.

Similarly, the two regex patterns explicitly referencing definitions - “can you define” and “definition of” - performed significantly better in the online Reddit corpus compared to the spoken corpora in BNC and Switchboard. “can you define” generated 263 matches in the BNC but only 7 WMNs, resulting in a low precision of 2.7%. Even lower numbers were found in Switchboard, where the regex generated 16 total matches but only 1 WMN (6.2% precision). However, in Reddit, it produced 574 matches and 141 confirmed WMNs, achieving a much higher precision of 24.6%. This strong performance highlights its alignment with the debate-oriented nature of online discourse, where explicit requests for definitions are more common than in spoken interaction. The regex “definition of” displays a similar pattern. In the BNC, it generated 95 matches with just 4 WMNs (4.2% precision), indicating limited use in meaningful negotiation within this corpus. Its presence in Switchboard was negligible, with only 2 matches and no confirmed WMNs. However, it performed markedly better in Reddit, where it produced 384 matches and 117 WMNs, achieving a precision of 30.5%. This suggests that “definition of” is particularly suited to capturing instances of semantic clarification and negotiation in text-based online interactions. Together, these patterns highlight the importance of context in determining regex effectiveness, with explicit mentions of definitions proving far more productive in online settings than in spoken interaction.

<table><caption>**Table 4** Regex success by corpus</caption><thead><tr><th rowspan="2">Regex Name</th><th colspan="3">BNC</th><th colspan="3">Switchboard</th><th colspan="3">Reddit</th></tr><tr><th># matches</th><th># WMN</th><th>% WMN</th><th># matches</th><th># WMN</th><th>% WMN</th><th># matches</th><th># WMN</th><th>% WMN</th></tr></thead><tbody><tr><td>you mean</td><td>424</td><td>61</td><td>14.4%</td><td>91</td><td>19</td><td>20.9%</td><td>319</td><td>92</td><td>28.8%</td></tr><tr><td>can you define</td><td>263</td><td>7</td><td>2.7%</td><td>16</td><td>1</td><td>6.2%</td><td>574</td><td>141</td><td>24.6%</td></tr><tr><td>definition of</td><td>95</td><td>4</td><td>4.2%</td><td>2</td><td>0</td><td>0.0%</td><td>384</td><td>117</td><td>30.5%</td></tr><tr><td>what is</td><td>812</td><td>35</td><td>4.3%</td><td>439</td><td>10</td><td>2.3%</td><td>233</td><td>20</td><td>8.6%</td></tr><tr><td>what do you mean by</td><td>61</td><td>23</td><td>37.7%</td><td>2</td><td>2</td><td>100.0%</td><td>91</td><td>38</td><td>41.8%</td></tr><tr><td>this is not X</td><td>809</td><td>8</td><td>1.0%</td><td>244</td><td>0</td><td>0.0%</td><td>876</td><td>51</td><td>5.8%</td></tr><tr><td>is that a/the</td><td>219</td><td>3</td><td>1.4%</td><td>263</td><td>1</td><td>0.4%</td><td>470</td><td>33</td><td>7.0%</td></tr><tr><td>this isn't X</td><td>264</td><td>1</td><td>0.4%</td><td>68</td><td>2</td><td>2.9%</td><td>541</td><td>33</td><td>6.1%</td></tr><tr><td>[repetition pattern]</td><td>888</td><td>26</td><td>2.9%</td><td>113</td><td>4</td><td>3.5%</td><td>5</td><td>0</td><td>0.0%</td></tr><tr><td>what X means</td><td>202</td><td>3</td><td>1.5%</td><td>11</td><td>0</td><td>0.0%</td><td>133</td><td>25</td><td>18.8%</td></tr><tr><td>the term</td><td>90</td><td>4</td><td>4.4%</td><td>11</td><td>0</td><td>0.0%</td><td>56</td><td>16</td><td>28.6%</td></tr><tr><td>in what way/sense</td><td>43</td><td>3</td><td>7.0%</td><td>5</td><td>0</td><td>0.0%</td><td>25</td><td>8</td><td>32.0%</td></tr><tr><td>what does X mean</td><td>96</td><td>9</td><td>9.4%</td><td>0</td><td>0</td><td>0.0%</td><td>12</td><td>2</td><td>16.7%</td></tr><tr><td>are you talking about</td><td>18</td><td>0</td><td>0.0%</td><td>16</td><td>3</td><td>18.8%</td><td>18</td><td>4</td><td>22.2%</td></tr><tr><td>what is the difference between</td><td>49</td><td>5</td><td>10.2%</td><td>5</td><td>2</td><td>40.0%</td><td>16</td><td>0</td><td>0.0%</td></tr></tbody></table>

The performance of regexes in identifying WMNs varies significantly depending on the type of WMN they capture: NONs (WMNs originating in insufficient/non-understanding of word meaning) and DINs (WMNs arising from disagreement about what a word can or should mean), which is displayed in Table 5. Regexes such as “you mean”, “what do you mean by” and “what is” are especially effective at identifying NONs. These patterns often appear in contexts where participants request clarification to address a lack of understanding or to seek further
explanation of a word's meaning. Their frequent association with NONs suggests that these regexes are well-suited for detecting sequences where comprehension issues are the driving force of the negotiation.

Conversely, regexes such as “can you define” and “definition of” are more strongly associated with DINs, capturing interaction sequences where participants are in disagreement about word meanings and need to draw upon definitions to make their case about what a word actually means. As a result, these patterns tend to appear in discussions that involve challenges to interpretations or efforts to redefine words, making them especially relevant in contexts of semantic negotiation.

In addition, the regexes “this is not X” and “this isn’t X” stand out for their strong association with DINs, with almost all of the matches falling into this category. Although these patterns did not produce a high number of total matches, their precision in identifying DINs is exceptionally high, reflecting their specialized role in capturing sequences of negotiation of word meaning originating in disagreement. This connection is not surprising, as these phrases are commonly used as meta-linguistic objections initiating DINs. Such objections occur when a participant challenges the appropriateness or validity of a word used by another participant, asserting that the term does not fit the context or intended meaning. Overall, the strong connection between these regexes and DINs highlights their importance in identifying disagreement-driven negotiations of meaning, making them valuable tools for studying these specific types of interactional phenomena.

<table><caption>**Table 5** Regex success in relation to WMN type</caption><thead><tr><th>Regex Name</th><th># WMNs</th><th># NONs</th><th># DINs</th><th># Other</th><th>% NONs</th><th>% DINs</th><th>% Other</th></tr></thead><tbody><tr><td>you mean</td><td>172</td><td>107</td><td>45</td><td>20</td><td>62.2%</td><td>26.2%</td><td>11.6%</td></tr><tr><td>can you define</td><td>149</td><td>42</td><td>107</td><td>0</td><td>28.2%</td><td>71.8%</td><td>0.0%</td></tr><tr><td>definition of</td><td>121</td><td>11</td><td>110</td><td>0</td><td>9.1%</td><td>90.9%</td><td>0.0%</td></tr><tr><td>what is</td><td>65</td><td>46</td><td>18</td><td>1</td><td>70.8%</td><td>27.7%</td><td>1.5%</td></tr><tr><td>what do you mean by</td><td>63</td><td>46</td><td>16</td><td>1</td><td>73.0%</td><td>25.4%</td><td>1.6%</td></tr><tr><td>this is not X</td><td>59</td><td>3</td><td>55</td><td>1</td><td>5.1%</td><td>93.2%</td><td>1.7%</td></tr><tr><td>is that a/the</td><td>37</td><td>9</td><td>28</td><td>0</td><td>24.3%</td><td>75.7%</td><td>0.0%</td></tr><tr><td>this isn't X</td><td>36</td><td>7</td><td>29</td><td>0</td><td>19.4%</td><td>80.6%</td><td>0.0%</td></tr><tr><td>[repetition pattern]</td><td>30</td><td>29</td><td>0</td><td>1</td><td>96.7%</td><td>0.00%</td><td>3.3%</td></tr><tr><td>what X means</td><td>28</td><td>5</td><td>23</td><td>0</td><td>17.9%</td><td>82.1%</td><td>0.0%</td></tr><tr><td>the term</td><td>20</td><td>4</td><td>16</td><td>0</td><td>20.0%</td><td>80.0%</td><td>0.0%</td></tr><tr><td>in what way/sense</td><td>11</td><td>3</td><td>8</td><td>0</td><td>27.3%</td><td>72.7%</td><td>0.0%</td></tr><tr><td>what does X mean</td><td>11</td><td>10</td><td>1</td><td>0</td><td>90.9%</td><td>9.1%</td><td>0.0%</td></tr><tr><td>are you talking about</td><td>7</td><td>4</td><td>1</td><td>2</td><td>57.1%</td><td>14.3%</td><td>28.6%</td></tr><tr><td>what is the difference between</td><td>7</td><td>6</td><td>1</td><td>0</td><td>85.7%</td><td>14.3%</td><td>0.0%</td></tr></tbody></table>

The use of regex patterns in identifying WMNs has proven effective but also highlights areas for refinement. Future studies aiming to build more robust WMN corpora more efficiently should consider both the strengths and limitations observed in current regex performance. Regexes such as “you mean” and “what do you mean by” illustrate the importance of balancing specificity and coverage. Regexes such as “what is”, “this is not X” and the repetition pattern capture valuable WMNs but are overly broad, leading to many irrelevant matches. To address this, future studies could analyze positive and negative cases, i.e., look more closely at both the successfully identified WMNs and the irrelevant matches produced. Identifying commonalities in the negative cases could
perhaps lead to conclusions that could inform the creation of more targeted patterns, leading to less irrelevant matches and more successful instances.

We carried out this analysis for cases matching the repetition pattern and compiled a list of 34 expressions such as “Is it?” or “Really?” which were almost never part of an interesting phenomenon. We recalculated its success rate after excluding all instances with these expressions. The percentage of successfully captured WMNs went from 3 to 3.7%, while that of total successful matches increased from 3.4% to 4.2%. While this may not seem like a substantial increase, and one WMN is lost in the process, 220 instances that were annotated as NOTHING would have been omitted. This is an example of an easy-to-implement improvement that can help speed up the annotation process.

Other improvement methods could involve the addition of contextual constraints, such as multi-turn patterns that take into consideration the turn following a potential indicator. A regex-based approach could also be complemented with machine learning techniques, using the annotated data to train models to classify sequences more accurately.

# 6. Inter-Annotator Agreement

In this section, we present our inter-annotator agreement (IAA) study. Three annotators were recruited and trained to label a sample of the corpus data. We then evaluated their agreement with one another as well as with our reference annotations.

## 6.1 Method

### 6.1.1 Annotation phases

We involved three female Master's students in Computational Linguistics from the University of Gothenburg to annotate a subset of the corpus in order to evaluate the complexity of the annotation task and the clarity of our guidelines. The annotators were native speakers of Polish, Brazilian Portuguese, and Finnish, all with an advanced level of English and fluent speaking skills. This step was crucial to assess how well-defined the task is, and to estimate the reliability that could be expected from the annotation process. The process was divided into three stages: two training phases and the annotation of a sample of the corpus.

**Training 1: Preparation and learning**

The annotators were introduced to the task through the written guidelines and two training videos. These videos covered the concept of WMN, the use of the Label Studio annotation platform, and included a few illustrative examples. Following this training, we held a joint meeting with all annotators to address any questions or concerns they had on the material before they proceeded to annotate independently.

**Training 2: Practice and refinement**

In this phase, the annotators worked through two rounds of practice annotations, each consisting of 15 examples. The training samples comprised a total of ten NONs, five DINs, two OTHER KINDS OF CLARIFICATION REQUESTS, one NON-PURSUED sequence and 12 NOTHING. After each round, we provided personalized feedback based on the expected annotations. This process also resulted in enhancements to the guidelines, clarifying areas that had been unclear during the practice annotations.

**Main annotation phase**

Once training was complete, annotators proceeded to annotate the sample with no further feedback. All three worked on the same data to allow for inter-annotator agreement analysis. No additional instructions were given,
except for resolving technical or procedural issues. This hands-off approach ensured that the annotations reflected the annotators' independent understanding of the task.

## 6.1.2 Sample composition

Table 6 presents the composition of the sample annotated by the three annotators, which consists of 704 instances: 305 from the BNC, 277 from Reddit and 122 from Switchboard. We based our sample selection on the distribution of labels related to the phenomena of interest within our corpus and across the three data sources. To decide the number of phenomena to include, we slightly adjusted proportions according to two criteria: for rare phenomena, such as WITHOUT TRIGGER, we included all available cases to ensure the most reliable agreement estimation possible. We also prioritized including complete WMNs over incomplete ones, as they are the focus of our study. After establishing the sample composition, we randomly sampled the number of instances that had been decided upon for each label from the corpus. Finally, we completed the sample by adding the same number of randomly sampled NOTHING sequences as there were instances for all other labels together. The sample size was determined by the anticipated time commitment for the annotators, calculated based on their pace during the training phase. It took the annotators between 57 and 65.5 hours to complete the main annotation phase, which corresponds to 0.18 to 0.2 instances per minute. Notably, this sample is much more dense in terms of interesting (not NOTHING) cases compared to the entire NeWMe corpus. Additionally, the annotators received instances in a random sequence, which likely slowed down the annotation process compared to the reference. In the reference setup, NOTHING cases were more common and instances from the same conversation were presented sequentially, offering more context and clarity.

<table><caption>**Table 6** Composition of the sample for student annotators</caption><thead><tr><th>Expert-annotated label</th><th>BNC</th><th>Reddit</th><th>Switchboard</th><th>Total</th></tr></thead><tbody><tr><td>NONs</td><td>53</td><td>28</td><td>16</td><td>97</td></tr><tr><td>DINs</td><td>10</td><td>62</td><td>0</td><td>72</td></tr><tr><td>WMN: Other</td><td>14</td><td>3</td><td>3</td><td>20</td></tr><tr><td>Non-pursued</td><td>3</td><td>51</td><td>1</td><td>55</td></tr><tr><td>SIMN</td><td>26</td><td>2</td><td>3</td><td>31</td></tr><tr><td>Without trigger</td><td>10</td><td>0</td><td>2</td><td>12</td></tr><tr><td>Reference/NE</td><td>6</td><td>3</td><td>16</td><td>25</td></tr><tr><td>Other kinds of clarification requests</td><td>4</td><td>25</td><td>11</td><td>40</td></tr><tr><td>Total relevant phenomena</td><td>126</td><td>174</td><td>52</td><td>352</td></tr><tr><td>Nothing</td><td>179</td><td>103</td><td>70</td><td>352</td></tr></tbody></table>

## 6.1.3 Agreement metrics

**Label** agreement

We verify the extent to which annotators agree with each other and with the reference regarding the phenomenon (or, in a few cases, phenomena) present in an instance. We calculate this for the raw annotations, as well as at the level of individual labels and sets of labels within specific categories: WMN (NONs, DINs and Other) and WMN-like (which refers to the combination of WMNs and incomplete WMNs: NON-PURSUED, SIMN and WITHOUT TRIGGER). To assess this, we use **Krippendorff's alpha** (Krippendorff, 2013) because of its flexibility in handling the number of annotators, possible values and types of data. This metric ranges from -1 to 1, with 1 indicating perfect agreement between annotators. This agreement is calculated over all available instances. Since some instances were annotated with more than one phenomenon, we modified Krippendorff's alpha difference function
to be the inverse of the overlap in a set of labels (i.e., if one annotator found two phenomena in an instance, e.g., a NON-PURSUED and a NON, and the other annotator only found the NON, the difference metric would be 0.5, reflecting the partial agreement between the two annotators).

## Span agreement

When annotators agree that a phenomenon is present, we measure their agreement on the location of its components (trigger, indicator and negotiation). Here, we quantify the annotators’ agreement with each other and with the reference on each kind of span label. Note that this agreement is only relevant when the spans are present; that is, when the annotators agree that the instance in question contains a phenomenon requiring span annotation. Consequently, for this calculation, we include only instances where both annotators marked the relevant spans. For **triggers**, we calculate two measures: the **exact span match ratio** and the **string overlap ratio**, which we report averaged across instances. The first measure takes into account the exact location of the triggers annotated, down to the character level. The second measure calculates the overlap in the sets of words annotated as triggers. These two measures are complementary. For example, if two annotators found a phenomenon related to the word “tea,” but they annotated the trigger in different utterances, the exact span match ratio will be 0 but the string overlap ratio will be 1, because they agreed on the actual problematic word, but not on its exact location. Conversely, if they agreed on the trigger’s location but one of them annotated it as “green tea” instead of “tea,” the string overlap ratio would be 0, but the exact span match ratio would be 1/3.

For indicators and negotiations, we do not evaluate exact span overlap, but instead focus on agreement at the utterance level. For the **indicator**, we report the percentage of phenomena where both annotators marked the indicator in the same utterance. For the **negotiation**, which can span multiple turns, we calculate the **F1-score** at the utterance level, which measures the extent to which two annotators agree on which utterances of the conversation contained a negotiation span.<fnref n="15" /> We report the average F1-score of all instances compared.

## Agreement on type of meaning

We calculate **Krippendorff’s alpha** to assess the agreement on the type of meaning discussed in a WMN. This calculation is restricted to instances where annotators agreed on the presence of a WMN. We treat the annotation of “both” types of meaning as a double annotation of “situated” and “potential” meaning instead of as a separate third label, and we use the inverse overlap metric as done for the label agreement.

## 6.2 Results

In this section we present all inter-annotator agreement results. We refer to the three annotators as A1, A2 and A3, and to the expert reference annotation as R.

### 6.2.1 Label agreement

We begin our analysis by examining label agreement across different label groups and on the raw annotation (Table 7) and each individual label (Table 8). The highest global agreement (A1-A2-A3-R) was observed for the NOTHING label, indicating that determining whether there is some interesting phenomenon in an instance or not was the easiest task to agree upon. Interestingly, NONs display a much higher agreement than DINs (0.55 vs 0.35). While this could suggest that DINs are more challenging to annotate consistently, it may also reflect corpus-specific effects, which we describe below.

Agreement varied considerably between annotators. Overall, A3 was the most closely aligned with the reference annotations, achieving up to 0.67 agreement for WMN-like phenomena. Krippendorff’s alpha values are often considered to indicate moderate agreement starting at 0.68, reflecting the global difficulty and subjectivity of the

<footnote n="15">¹⁵ For example, if annotator A1 marked turns 4 and 5 as part of the negotiation and annotator A2 marked turns 5, 6 and 7, A2’s precision with respect to A1 is ¹/₃, recall is ½, and F1 is their harmonic mean, 0.4.</footnote>
task. However, A3's performance, approaching this threshold, suggests that moderate agreement is attainable for certain distinctions, specifically for identifying complete or incomplete WMNs.

<table><thead><tr><th></th><th>Annotators</th><th>WMN</th><th>WMN-like</th><th>raw annotation</th></tr></thead><tbody><tr><td rowspan="4">Table 7</td><td>A1-R</td><td>0.51</td><td>0.60</td><td>0.56</td></tr><tr><td>A2-R</td><td>0.63</td><td><strong>0.65</strong></td><td>0.55</td></tr><tr><td>A3-R</td><td>0.64</td><td><strong>0.68</strong></td><td>0.60</td></tr><tr><td>A1-A2-A3</td><td>0.53</td><td>0.59</td><td>0.51</td></tr><tr><td></td><td>A1-A2-A3-R</td><td>0.56</td><td>0.62</td><td>0.54</td></tr></tbody></table>

Krippendorff's alpha for groups of phenomena (WMN and WMN-like) and for the raw annotation

<table><caption>**Table 8** Krippendorff's alpha by individual label</caption><thead><tr><th></th><th>NON</th><th>DIN</th><th>WMN: Other</th><th>SIMN</th><th>Non-pursued</th><th>Without trigger</th><th>Reference/ NE</th><th>Other kinds of clarification requests</th><th>Nothing</th></tr></thead><tbody><tr><td>A1-R</td><td>0.58</td><td>0.20</td><td>0.53</td><td>0.53</td><td>0.52</td><td>0.70</td><td>0.49</td><td>0.47</td><td>0.63</td></tr><tr><td>A2-R</td><td>0.56</td><td>0.51</td><td>0.43</td><td>0.52</td><td>0.54</td><td>0.53</td><td>0.20</td><td>0.37</td><td><strong>0.65</strong></td></tr><tr><td>A3-R</td><td>0.64</td><td>0.43</td><td>0.41</td><td>0.55</td><td>0.53</td><td><strong>0.66</strong></td><td>0.40</td><td>0.51</td><td><strong>0.66</strong></td></tr><tr><td>A1-A2-A3</td><td>0.50</td><td>0.28</td><td>0.29</td><td>0.53</td><td>0.51</td><td>0.53</td><td>0.14</td><td>0.47</td><td>0.61</td></tr><tr><td>A1-A2-A3-R</td><td>0.55</td><td>0.35</td><td>0.37</td><td>0.53</td><td>0.52</td><td>0.58</td><td>0.27</td><td>0.46</td><td>0.63</td></tr></tbody></table>

We also calculated agreement separately for each corpus (Table 9). This analysis revealed a notable difference between Reddit and the spoken corpora. Agreement values are considerably higher for the spoken corpora, with A3 reaching 0.70 and 0.81 agreement with the reference on BNC and Switchboard. The other annotators also achieved values no lower than 0.64 for the WMN-like phenomena in these corpora. These results suggest that moderate agreement can be obtained for the general task of annotating WMN-like phenomena with the current training material. Further simplification of the guidelines and labelling scheme, along with more extensive training, could enhance consistency.

The lower agreement observed for the Reddit data partly explains the lower agreement values of DINs, which are almost exclusively found in that part of the corpus.<fnref n="16" /> Reddit data is clearly more challenging to annotate, with its multiple intertwined threads and lengthy posts. In future work, we should consider modifying the data presentation for annotation, for example displaying only the relevant subthread instead of the entire thread and visually clarifying the tree structure.

<table><thead><tr><th>BNC</th><th>Reddit</th><th>Switchboard</th></tr></thead></table>

<footnote n="16">¹⁶ While it is possible that DINs are also inherently more difficult to agree upon, we observe a tendency for Reddit data to display lower agreement also in NONs and Nothings, which are common phenomena in the three corpora.</footnote>
<table><caption>**Table 9** Krippendorff's alpha for groups of phenomena, broken down by corpus</caption><thead><tr><th>Annotators</th><th>WMN</th><th>WMN-like</th><th>WMN</th><th>WMN-like</th><th>WMN</th><th>WMN-like</th></tr></thead><tbody><tr><td>A1-R</td><td>0.63</td><td><strong>0.68</strong></td><td>0.35</td><td>0.48</td><td>0.62</td><td><strong>0.67</strong></td></tr><tr><td>A2-R</td><td><strong>0.68</strong></td><td>0.64</td><td>0.57</td><td>0.63</td><td>0.61</td><td><strong>0.65</strong></td></tr><tr><td>A3-R</td><td><strong>0.66</strong></td><td><strong>0.70</strong></td><td>0.57</td><td>0.58</td><td><strong>0.79</strong></td><td><strong>0.81</strong></td></tr><tr><td>A1-A2-A3</td><td>0.63</td><td><strong>0.68</strong></td><td>0.41</td><td>0.46</td><td>0.53</td><td>0.61</td></tr><tr><td>A1-A2-A3-R</td><td><strong>0.65</strong></td><td><strong>0.68</strong></td><td>0.46</td><td>0.52</td><td>0.60</td><td><strong>0.66</strong></td></tr></tbody></table>

A closer look at the confusion matrices (Figures 2, 3 and 4 in the Appendix) reveals interesting insights into the nature of the differences in annotation. While each annotator exhibits specific tendencies, we observe two frequent patterns: (1) instances labeled as DINs in the reference are often annotated as NOTHING, and (2) the labels WMN: OTHER and NON are often confused with one another. Similarly, REFERENCE/NE is often annotated as NON or OTHER KINDS OF CLARIFICATION REQUESTS. Moreover, A1 labeled a large proportion of the sample as NOTHING. A larger training sample could have helped identify these patterns of inconsistency earlier, allowing us to provide more personalized feedback or refine the guidelines to clarify distinctions between the commonly confused classes. However, the limited availability of data for certain labels restricted the size and composition of the training sample.

Finally, we present results of the agreement on the type of meaning in Table 10. The agreement values attested for this label range from 0.46 to 0.66. As we observed during the reference annotation process, this is quite a subjective notion that is hard to agree upon. In future annotation projects/efforts including the type of meaning, it may be necessary to provide more training dedicated to this notion.

<table><caption>**Table 10** Krippendorff's alpha for the annotation of the type of meaning</caption><thead><tr><th>Annotators</th><th>Krippendorff's alpha</th></tr></thead><tbody><tr><td>A1-R</td><td>0.52 (99)</td></tr><tr><td>A2-R</td><td>0.46 (167)</td></tr><tr><td>A3-R</td><td>0.59 (143)</td></tr><tr><td>A1-A2</td><td>0.62 (108)</td></tr><tr><td>A1-A3</td><td>0.66 (93)</td></tr><tr><td>A2-A3</td><td>0.51 (146)</td></tr></tbody></table>

### 6.2.2 Span agreement

In Table 11, we present results of span-related agreement. Once annotators agree that an instance contains a phenomenon requiring a specific kind of span, the agreement on the location of these spans shows higher consistency. Match ratios for trigger spans are between 0.64 and 0.78, indicating a moderate but acceptable agreement. For indicators, agreement is even higher, with indicators being located in the same utterance as the reference between 84% and 90% of the time. We expected indicators to show a high agreement, because they tend to be the most easily identifiable part of a WMN. Finally, F1-scores for negotiation spans are all above 0.72, indicating more than acceptable agreement.
Agreement on spans shows less variation across annotators compared to other measures, and agreement values with the reference annotations are comparable to those observed between pairs of annotators. This suggests that the guidelines for span identification are mostly clear and that, despite some degree of disagreement and subjectivity, identifying spans is a relatively straightforward task.

<table><caption>**Table 11** IAA results on trigger, indicator and negotiation spans.</caption><thead><tr><th rowspan="2">Annotators</th><th colspan="2">Trigger</th><th>Indicator</th><th>Negotiation</th></tr><tr><th>Exact span match ratio</th><th>String match ratio (# comparisons)</th><th>% of coincidence (# comparisons)</th><th>F1-score (# comparisons)</th></tr></thead><tbody><tr><td>A1-R</td><td>0.73</td><td>0.74 (222)</td><td>90% (261)</td><td>0.78 (134)</td></tr><tr><td>A2-R</td><td>0.69</td><td>0.77 (296)</td><td>87% (347)</td><td>0.73 (206)</td></tr><tr><td>A3-R</td><td>0.71</td><td>0.76 (273)</td><td>84% (315)</td><td>0.75 (186)</td></tr><tr><td>A1-A2</td><td>0.69</td><td>0.75 (233)</td><td>89% (276)</td><td>0.76 (144)</td></tr><tr><td>A1-A3</td><td>0.70</td><td>0.78 (220)</td><td>85% (265)</td><td>0.77 (131)</td></tr><tr><td>A2-A3</td><td>0.64</td><td>0.69 (296)</td><td>78% (350)</td><td>0.72 (191)</td></tr></tbody></table>

## 7. Conclusion

We have presented the NeWMe Corpus, the first corpus dedicated to the analysis of Word Meaning Negotiation (WMN). The NeWMe Corpus contains manual annotations for a total of 404 WMN sequences as well as other related phenomena, including incomplete WMNs and distractors.

Our analysis of the annotated data revealed several interesting tendencies regarding the prevalence of each phenomenon across different forms of interaction and activity types. Our annotation of the type of meaning showed that, while most WMNs center on situated meaning, DINs tend to discuss potential meaning more often than NONs. This distinction underscores the different functions of these negotiation types: NONs primarily aim at resolving immediate comprehension issues, while DINs often involve broader debates on word definitions. We also found that failed and abandoned attempts at initiating WMN (NON-PURSUED sequences) and negotiations arising from disagreements about word meaning are much more common in asynchronous online interaction than in spoken interaction. This pattern reflects the unique characteristics of each type of interaction. In asynchronous platforms like online forums, participants can take their time to respond or choose not to respond at all, making it easier for disagreements to drag on or for clarification attempts to be ignored. In contrast, spoken interactions occur in real-time, where the flow of conversation relies on immediate turn-taking and sequential progression. However, it remains unclear to what extent these differences are driven by the type of interaction activity rather than the mode of the interaction itself. The online data in our corpus mainly consists of open-ended discussions, while the spoken data includes a greater variety of interactional contexts, including interviews, meetings and informal discussions. Open-ended discussions may naturally involve more disagreement, contributing to the
higher prevalence of DINs in the online data. Future research could aim at disentangling the role of activity type from the interaction mode in shaping WMN frequency and characteristics.

While the agreement in the reference annotation is high, our IAA analysis with three external annotators revealed areas of improvement. The IAA results broken down by corpus and label show that an acceptable level of agreement is attainable despite the subjectivity of the task, but that (1) the annotation setup and labels could be simplified, (2) written threaded online discussions such as those found in Reddit should be presented in a more streamlined format, and (3) more extensive and targeted annotator training should be provided to enhance consistency.

We also carried out an analysis of our regular expression-based methodology for candidate selection, which served to evaluate its efficiency and point to directions of improvement for future work. This analysis identified expressions that are highly indicative of a WMN, as well as others that require additional filtering or refinement to improve precision and reduce irrelevant matches. For example, while expressions such as “what do you mean by X?” had a high precision (40.9%), broader expressions like “what is” and “this is not X” had much lower precision (4.9% and 3.1%, respectively). Refining these regexes and supplementing them with machine learning-based filtering could improve both precision and recall in future iterations of corpus development, reducing annotation workload and enhancing the overall efficiency of data collection. We also observed differences depending on the type of WMN: indicators including “you mean” were more often involved in NONs, while expressions invoking the definition of a word (e.g., “can you define”) were mostly used in DINs.

The current approach does not allow for determining the true prevalence of WMNs in the selected corpora, as the analysis is limited to interactions captured by the regular expressions. The number of WMN instances using other expressions that may have been missed remains unknown, emphasizing the need for further methodological refinement and exploration. While our method successfully identified 404 WMNs, its recall remains unknown since we do not have a measure of the actual prevalence of WMNs in the corpora. Estimating recall would require manually reading through a large portion of the corpus, which is highly impractical for a phenomenon that appears relatively rare: we found WMNs in 6% of all conversations considered, which can be regarded as a lower bound of its frequency in the data used.

In future work, we plan to expand this corpus by including additional languages, conversational settings, and modalities. This extension will incorporate the lessons learned from this study, complementing regular expression searches with semi-automatic corpus expansion methods informed by our manual annotations. For example, we may find additional ways of expressing an indicator that we had not previously considered by looking into indicators that were annotated in the vicinity of a regex match. Additionally, we aim to explore machine learning approaches to enhance the identification of WMNs, leveraging the annotated data as training material. Future work could also explore dialogue acts and strategies for contributing to ongoing WMN sequences, drawing on Myrendal’s (2015) taxonomy of WMN-related dialogue acts, which includes mechanisms such as explicification, exemplification, and contrast.

The standoff annotations will be made available<fnref n="17" /> to the research community, accompanied by tools and instructions for retrieving the corpora and inspecting annotated interactions. We will also make available the complete annotation guidelines, enabling further research on WMN and related phenomena such as conversational repair and clarification requests. Furthermore, the annotated instances will be freely browsable online, providing an accessible platform for researchers and practitioners to explore the corpus and draw insights from its rich dataset. We will also provide the actual regular expressions used to match the relevant utterances in the corpora and the relevant scripts used for retrieving and analyzing data. By making these resources widely available, we hope to advance the empirical and theoretical study of WMNs and support the development of practical applications, such as adaptive dialogue systems capable of navigating and resolving potential semantic misalignments in real-time interaction.

<footnote n="17">17 https://github.com/gu-wmn/NeWMe/</footnote>
References

Babcock, M. J., Ta, V. P., & Ickes, W. (2014). Latent semantic similarity and language style matching in initial dyadic interactions. *Journal of Language and Social Psychology*, 33(3), 77–88.

Barnidge, M. (2017). Exposure to political disagreement in social media versus face-to-face and anonymous online settings. *Political Communication*, 34(2), 302–321. https://doi.org/10.1080/10584609.2016.1235639

BNC Consortium. (2007). *The British National Corpus, XML Edition*. Oxford Text Archive. http://hdl.handle.net/20.500.14106/2554

boyd, d. m., & Ellison, N. B. (2007). Social network sites: Definition, history, and scholarship. *Journal of Computer-Mediated Communication*, 13(1), 210–230. https://doi.org/10.1111/j.1083-6101.2007.00393.x

Branigan, H. P., Pickering M. J. & Cleland, A. A. (2000). Syntactic co-ordination in dialogue. *Cognition*, 75(2):B13–B25. https://doi.org/10.1016/S0010-0277(99)00081-5

Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. *Journal of Experimental Psychology: Learning, Memory, and Cognition*, 22(6), 1482–1493.

Chang, J. P., Chiam, C., Fu, L., Wang, A., Zhang, J., & Danescu-Niculescu-Mizil, C. (2020, July). ConvoKit: A toolkit for the analysis of conversations. In *Proceedings of the 21st Annual Meeting of the Special Interest Group on Discourse and Dialogue*.

Clark, H. H. (1996). *Using language*. Cambridge: Cambridge University Press.

Cleland, A. A., & Pickering, M. J. (2003). The use of lexical and syntactic information in language production: Evidence from the priming of noun-phrase structure. *Journal of Memory and Language*, 49(2), 214–230.

Drew, P. (1997). ‘Open’ class repair initiators in response to sequential sources of troubles in conversation. *Journal of Pragmatics*, 28(1), 69–101.

Ellison, N. B., Steinfield, C., & Lampe, C. (2007). The benefits of Facebook “friends:” Social capital and college students’ use of online social network sites. *Journal of Computer-Mediated Communication*, 12(4), 1143–1168. https://doi.org/10.1111/j.1083-6101.2007.00367.x

Garí Soler, A., Labeau, M., & Clavel, C. (2023). Measuring lexico-semantic alignment in debates with contextualized word representations. In *Proceedings of the First Workshop on Social Influence in Conversations (SICon 2023)* (pp. 50–63). Toronto, Canada. https://doi.org/10.18653/v1/2023.sicon-1.6

Garfinkel, H. (1967). *Studies in ethnomethodology*. Englewood Cliffs, NJ: Prentice-Hall.

Heritage, J. (1984). *Garfinkel and ethnomethodology*. Cambridge: Polity Press.

Hough, J., & Purver, M. (2013). Modelling expectations in the self-repair processing of annot-, um, listeners. In R. Fernández & A. Isard (Eds.), *Proceedings of the 17th SemDial Workshop on the Semantics and Pragmatics of Dialogue* (pp. 92–101). Amsterdam, Netherlands, December 16–18.

Karafoti, E. (2021). Negotiating preferred norms in requests and offers: Is the (dis)preferred answer so obviously (im)polite? *Journal of Pragmatics*, 173, 134–147. https://doi.org/10.1016/j.pragma.2020.12.012

Krippendorff, K. (2013). *Content analysis: An introduction to its methodology. 3rd edition*. Thousand Oaks, CA: Sage.
Larsson, S., & Myrendal, J. (2017a). Towards dialogue acts and updates for semantic coordination. *Proceedings of the Workshop on Formal Approaches to the Dynamics of Linguistic Interaction* (FADLI 2017).

Larsson, S., & Myrendal, J. (2017b). Dialogue acts and updates for semantic coordination. *Proceedings of the 21st Workshop on the Semantics and Pragmatics of Dialogue, Saarbrücken, 15–17 August 2017*. In V. Petukhova & Y. Tian (Eds.).

Long, M. H. (1996). The role of the linguistic environment in second language acquisition. In W. C. Ritchie & T. K. Bhatia (Eds.), *Handbook of research on language acquisition: Second language acquisition* (Vol. 2, pp. 413–468). New York: Academic Press.

Myrendal, J. (2015). *Word meaning negotiation in online discussion forum communication* (Doctoral dissertation). University of Gothenburg.

Myrendal, J. (2019). Negotiating meanings online: Disagreements about word meaning in discussion forum communication. *Discourse Studies, 21*(3), 1–23.

Noble, B., Viloria, K., Larsson, S., & Sayeed, A. (2021). What do you mean by negotiation? Annotating social media discussions about word meaning. In *Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue - Full Papers*.

Norén, K., & Linell, P. (2007). Meaning potentials and the interaction between lexis and contexts: An empirical substantiation. *Pragmatics, 17*(3), 387–416. https://doi.org/10.1075/prag.17.3.03nor

Pickering, M., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. *Behavioral and Brain Sciences, 27*(2), 169–226.

Pitzl, M.-L. (2005). Non-understanding in English as lingua franca: Examples from a business context. *Vienna English Working Papers, 14*(2), 50–71.

Pomerantz, A., & Heritage, J. (2013). Preference. In J. Sidnell & T. Stivers (Eds.), *The handbook of conversation analysis* (pp. 210–228). Malden, MA: Wiley-Blackwell.

Purver, M. (2004). *The theory and use of clarification requests in dialogue* (Ph.D. thesis). University of London.

Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. *Language, 50*(4), 696–735.

Schegloff, E. A. (2007). *Sequence organization in interaction: A primer in conversation analysis* (Vol. 1). Cambridge: Cambridge University Press.

Schober, M. F. (2005). Conceptual alignment in conversation. In S. T. Fiske, H. R. Markus, & P. A. Glick (Eds.), *Other minds: How humans bridge the divide between self and others* (pp. 239–252). New York, NY: Guilford Press.

Stivers, T., & Robinson, J. D. (2006). A preference for progressivity in interaction. *Language in Society, 35*(3), 367–392.

Stolk, A., Verhagen, L., & Toni, I. (2016). Conceptual alignment: How brains achieve mutual understanding. *Trends in Cognitive Sciences, 20*(3), 180–191. https://doi.org/10.1016/j.tics.2015.11.007

Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Van Ess-Dykema, C., & Meteer, M. (2000). Dialogue act modeling for automatic tagging and recognition of conversational speech. *Computational Linguistics, 26*(3), 339–373.
Tan, C., Niculae, V., Danescu-Niculescu-Mizil, C., & Lee, L. (2016, April). Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In *Proceedings of the 25th International Conference on World Wide Web* (pp. 613–624).

Varonis, E. M., & Gass, S. (1985). Non-native/non-native conversations: A model for negotiation of meaning. *Applied Linguistics*, 6*(1), 71–90.

Vasseur, M.-T., Broeder, P., & Roberts, C. (1996). Managing understanding from a minority perspective. In K. Bremer, C. Roberts, M.-T. Vasseur, M. Simonot, & P. Broeder (Eds.), *Achieving understanding: Discourse in intercultural encounters* (pp. 65–108). London: Longman.

## 8. Appendix

<figure><img src="image_7.png" /><figcaption>Fig 1 The LabelStudio annotation interface</figcaption></figure>

<table><thead><tr><th rowspan="2">Regex Name</th><th colspan="3">BNC</th><th colspan="3">Switchboard</th><th colspan="3">Reddit</th></tr><tr><th># matches</th><th># WMN</th><th>% WMN</th><th># matches</th><th># WMN</th><th>% WMN</th><th># matches</th><th># WMN</th><th>% WMN</th></tr></thead><tbody><tr><td>you mean</td><td>424</td><td>61</td><td>14.4%</td><td>91</td><td>19</td><td>20.9%</td><td>319</td><td>92</td><td>28.8%</td></tr><tr><td>can you define</td><td>263</td><td>7</td><td>2.7%</td><td>16</td><td>1</td><td>6.2%</td><td>574</td><td>141</td><td>24.6%</td></tr><tr><td>definition of</td><td>95</td><td>4</td><td>4.2%</td><td>2</td><td>0</td><td>0.0%</td><td>384</td><td>117</td><td>30.5%</td></tr><tr><td>what is</td><td>812</td><td>35</td><td>4.3%</td><td>439</td><td>10</td><td>2.3%</td><td>233</td><td>20</td><td>8.6%</td></tr><tr><td>what do you mean by</td><td>61</td><td>23</td><td>37.7%</td><td>2</td><td>2</td><td>100.0%</td><td>91</td><td>38</td><td>41.8%</td></tr><tr><td>this is not X</td><td>809</td><td>8</td><td>1.0%</td><td>244</td><td>0</td><td>0.0%</td><td>876</td><td>51</td><td>5.8%</td></tr><tr><td>is that a/the</td><td>219</td><td>3</td><td>1.4%</td><td>263</td><td>1</td><td>0.4%</td><td>470</td><td>33</td><td>7.0%</td></tr></tbody></table>
<table><caption>Table 12 Complete table of regex success by corpus</caption><tr><td>this isn't X</td><td>264</td><td>1</td><td>0.4%</td><td>68</td><td>2</td><td>2.9%</td><td>541</td><td>33</td><td>6.1%</td></tr><tr><td>[repetition pattern]</td><td>888</td><td>26</td><td>2.9%</td><td>113</td><td>4</td><td>3.5%</td><td>5</td><td>0</td><td>0.0%</td></tr><tr><td>what X means</td><td>202</td><td>3</td><td>1.5%</td><td>11</td><td>0</td><td>0.0%</td><td>133</td><td>25</td><td>18.8%</td></tr><tr><td>the term</td><td>90</td><td>4</td><td>4.4%</td><td>11</td><td>0</td><td>0.0%</td><td>56</td><td>16</td><td>28.6%</td></tr><tr><td>in what way/sense</td><td>43</td><td>3</td><td>7.0%</td><td>5</td><td>0</td><td>0.0%</td><td>25</td><td>8</td><td>32.0%</td></tr><tr><td>what does X mean</td><td>96</td><td>9</td><td>9.4%</td><td>0</td><td>0</td><td>0.0%</td><td>12</td><td>2</td><td>16.7%</td></tr><tr><td>are you talking about</td><td>18</td><td>0</td><td>0.0%</td><td>16</td><td>3</td><td>18.8%</td><td>18</td><td>4</td><td>22.2%</td></tr><tr><td>what is the difference between</td><td>49</td><td>5</td><td>10.2%</td><td>5</td><td>2</td><td>40.0%</td><td>16</td><td>0</td><td>0.0%</td></tr><tr><td>isn't this...?</td><td>33</td><td>0</td><td>0.0%</td><td>0</td><td>0</td><td>0.0%</td><td>45</td><td>7</td><td>15.6%</td></tr><tr><td>what sort/kind of</td><td>100</td><td>5</td><td>5.0%</td><td>25</td><td>1</td><td>4.0%</td><td>7</td><td>0</td><td>0.0%</td></tr><tr><td>elaborate</td><td>41</td><td>2</td><td>4.9%</td><td>13</td><td>0</td><td>0.0%</td><td>65</td><td>4</td><td>6.2%</td></tr><tr><td>the meaning of</td><td>15</td><td>1</td><td>6.7%</td><td>1</td><td>0</td><td>0.0%</td><td>35</td><td>4</td><td>11.4%</td></tr><tr><td>is this not...?</td><td>9</td><td>0</td><td>0.0%</td><td>4</td><td>0</td><td>0.0%</td><td>22</td><td>4</td><td>18.2%</td></tr><tr><td>are we talking</td><td>26</td><td>0</td><td>0.0%</td><td>3</td><td>1</td><td>33.3%</td><td>9</td><td>2</td><td>22.2%</td></tr><tr><td>do you know what X is</td><td>26</td><td>1</td><td>3.8%</td><td>9</td><td>0</td><td>0.0%</td><td>8</td><td>1</td><td>12.5%</td></tr><tr><td>is this what you mean</td><td>12</td><td>1</td><td>8.3%</td><td>1</td><td>0</td><td>0.0%</td><td>4</td><td>1</td><td>25.0%</td></tr><tr><td>word X means</td><td>18</td><td>1</td><td>5.6%</td><td>0</td><td>0</td><td>0.0%</td><td>10</td><td>1</td><td>10.0%</td></tr><tr><td>have you heard of</td><td>20</td><td>0</td><td>0.0%</td><td>10</td><td>1</td><td>10.0%</td><td>10</td><td>0</td><td>0.0%</td></tr><tr><td>is that what X is</td><td>11</td><td>0</td><td>0.0%</td><td>3</td><td>0</td><td>0.0%</td><td>1</td><td>1</td><td>100.0%</td></tr><tr><td>does X count as Y</td><td>1</td><td>0</td><td>0.0%</td><td>0</td><td>0</td><td>0.0%</td><td>5</td><td>1</td><td>20.0%</td></tr><tr><td>how do you mean that</td><td>0</td><td>0</td><td>0.0%</td><td>2</td><td>0</td><td>0.0%</td><td>0</td><td>0</td><td>0.0%</td></tr><tr><td>that would mean</td><td>19</td><td>0</td><td>0.0%</td><td>3</td><td>0</td><td>0.0%</td><td>26</td><td>0</td><td>0.0%</td></tr><tr><td>is that what that/this is</td><td>0</td><td>0</td><td>0.0%</td><td>1</td><td>0</td><td>0.0%</td><td>0</td><td>0</td><td>0.0%</td></tr></table>

<table><thead><tr><th>Regex Name</th><th># WMNs</th><th># NONs</th><th># DINs</th><th># Other</th><th>% NONs</th><th>% DINs</th><th>% Other</th></tr></thead><tbody><tr><td>you mean</td><td>172</td><td>107</td><td>45</td><td>20</td><td>62.2%</td><td>26.2%</td><td>11.6%</td></tr><tr><td>can you define</td><td>149</td><td>42</td><td>107</td><td>0</td><td>28.2%</td><td>71.8%</td><td>0.0%</td></tr><tr><td>definition of</td><td>121</td><td>11</td><td>110</td><td>0</td><td>9.1%</td><td>90.9%</td><td>0.0%</td></tr><tr><td>what is</td><td>65</td><td>46</td><td>18</td><td>1</td><td>70.8%</td><td>27.7%</td><td>1.5%</td></tr><tr><td>what do you mean by</td><td>63</td><td>46</td><td>16</td><td>1</td><td>73.0%</td><td>25.4%</td><td>1.6%</td></tr><tr><td>this is not X</td><td>59</td><td>3</td><td>55</td><td>1</td><td>5.1%</td><td>93.2%</td><td>1.7%</td></tr><tr><td>is that a/the</td><td>37</td><td>9</td><td>28</td><td>0</td><td>24.3%</td><td>75.7%</td><td>0.0%</td></tr><tr><td>this isn't X</td><td>36</td><td>7</td><td>29</td><td>0</td><td>19.4%</td><td>80.6%</td><td>0.0%</td></tr><tr><td>[repetition pattern]</td><td>30</td><td>29</td><td>0</td><td>1</td><td>96.7%</td><td>0.00%</td><td>3.3%</td></tr></tbody></table>
<table><caption>**Table 13** Complete table of regex success in relation to WMN type</caption><tr><td>what X means</td><td>28</td><td>5</td><td>23</td><td>0</td><td>17.9%</td><td>82.1%</td><td>0.0%</td></tr><tr><td>the term</td><td>20</td><td>4</td><td>16</td><td>0</td><td>20.0%</td><td>80.0%</td><td>0.0%</td></tr><tr><td>in what way/sense</td><td>11</td><td>3</td><td>8</td><td>0</td><td>27.3%</td><td>72.7%</td><td>0.0%</td></tr><tr><td>what does X mean</td><td>11</td><td>10</td><td>1</td><td>0</td><td>90.9%</td><td>9.1%</td><td>0.0%</td></tr><tr><td>are you talking about</td><td>7</td><td>4</td><td>1</td><td>2</td><td>57.1%</td><td>14.3%</td><td>28.6%</td></tr><tr><td>what is the difference between</td><td>7</td><td>6</td><td>1</td><td>0</td><td>85.7%</td><td>14.3%</td><td>0.0%</td></tr><tr><td>isn't this...?</td><td>7</td><td>3</td><td>4</td><td>0</td><td>42.9%</td><td>57.1%</td><td>0.0%</td></tr><tr><td>what sort/kind of</td><td>6</td><td>6</td><td>0</td><td>0</td><td>100.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>elaborate</td><td>6</td><td>5</td><td>1</td><td>0</td><td>83.3%</td><td>16.7%</td><td>0.0%</td></tr><tr><td>the meaning of</td><td>5</td><td>1</td><td>4</td><td>0</td><td>20.0%</td><td>80.0%</td><td>0.0%</td></tr><tr><td>is this not...?</td><td>4</td><td>1</td><td>3</td><td>0</td><td>25.0%</td><td>75.0%</td><td>0.0%</td></tr><tr><td>are we talking</td><td>3</td><td>3</td><td>0</td><td>0</td><td>100.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>do you know what X is</td><td>2</td><td>1</td><td>1</td><td>0</td><td>50.0%</td><td>50.0%</td><td>0.0%</td></tr><tr><td>is this what you mean</td><td>2</td><td>1</td><td>1</td><td>0</td><td>50.0%</td><td>50.0%</td><td>0.0%</td></tr><tr><td>word X means</td><td>2</td><td>0</td><td>2</td><td>0</td><td>0.0%</td><td>100.0%</td><td>0.0%</td></tr><tr><td>have you heard of</td><td>1</td><td>1</td><td>0</td><td>0</td><td>100.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>is that what X is</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0.0%</td><td>100.0%</td><td>0.0%</td></tr><tr><td>does X count as Y</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0.0%</td><td>100.0%</td><td>0.0%</td></tr><tr><td>how do you mean that</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td><td></td><td></td></tr><tr><td>that would mean</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td><td></td><td></td></tr><tr><td>is that what that/this is</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td><td></td><td></td></tr><tr><td>depends on what you mean</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td><td></td><td></td></tr></table>
<figure><img src="image_8.png" /><figcaption>Fig 2 Confusion matrix between the reference annotation and A1's annotation.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig 3 Confusion matrix between the reference annotation and A2's annotation.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Fig 4 Confusion matrix between the reference annotation and A3's annotation.</figcaption></figure>