KnowComp at DialAM-2024: Fine-tuning Pre-trained Language Models for Dialogical Argument Mining with Inference Anchoring Theory (2024)

Abstract: In this paper, we present our framework for DialAM-2024 Task A: Identification of Propositional Relations and Task B: Identification of Illocutionary Relations. The goal of Task A is to detect argumentative relations between propositions in an argumentative dialogue (Inference, Conflict, Rephrase), while Task B while Task B aims to detect illocutionary relations between locutions and argumentative propositions in a dialogue, e.g.„ Asserting, Agreeing, Arguing, Disagreeing. Noticing the definition of the relations are strict and professional under the context of IAT framework, we meticulously curate prompts which not only incorporate formal definition of the relations, but also exhibit the subtle differences between them. The PTLMs are then fine-tuned on the human-designed prompts to enhance its discrimination capability in classifying different theoretical relations by learning from the human instruction and the ground truth samples. After extensive experiments, a fine-tuned DeBERTa-v3-base model exhibits the best performance among all PTLMs with an F1 score of 78.90% on Task B. It is worth noticing that our framework ranks #2 in the ILO General official leaderboard.

heading: 1 Introduction
text: Dialogical argument mining is an emerging field that aims to bridge the gap between the analysis of argumentation and dialogue (Budzynska et al., 2014b; Ruiz-Dolz et al., 2024; Kawarada et al., 2024). Traditional argument mining approaches have often focused on opinion mining within monological texts (Lawrence and Reed, 2019; Arumugam, 2022) or document form contents (Ruosch et al., 2022; Sazid and Mercer, 2022; Khondoker and Yousuf, 2022). However, real-world argumentation frequently occurs in dialogical contexts, where multiple participants engage in a dy-
*Equal Contributions

heading: YA nodes
text: namic exchange of viewpoints (Feger and Dietze, 2024; Lai et al., 2024; Alsinet et al., 2022). This complexity necessitates a more holistic approach that considers both the argumentative structures and the dialogical interactions.
Apart from the dialogical information extraction paradigms explored by previous works (Dutta et al., 2022; Mestre et al., 2021), A generic modelling formalism for extracting dialogical information is the Inference Anchoring Theory (IAT) introduced by Budzynska and Reed (2011). It offers a systematic approach to decomposing text speech into distinct units (ADUs), while also anchoring and categorising logical inferences between propositions and locutions. As such, IAT provides a comprehensive methodology for analyzing the maneuvers of dialogues within a given theoretical framework, thus building an explicit scaffolding for language models to handle semantics analysis tasks (Budzynska et al., 2014a).
Based on this theory, DialAM-2024 workshop (Ruiz-Dolz et al., 2024) introduces the first shared task in dialogical argument mining, aimed at modeling argumentation and dialogue information together within a domain-independent framework. The proposed tasks of DialAM-2024 involves classification of the three-way argumentative relations between locutions and corresponding propositions,
103
detection of relevant dialogical components and completion of the inference anchoring map.
Due to the in-context learning ability of LLMs on unconventional tasks with demonstrated examples (Sun et al., 2023), our initial attempt was to use Large Language Models (LLMs) as the classifier for illocutionary relations (Chan et al., 2024; Wang et al., 2023b,a, 2024a,b; Wang and Song, 2024). A combination of zero-shot and few-shot (Brown et al., 2020) prompts integrated with Chainof-thought (Wei et al., 2022) were tested. However, we observed that popular LLMs, such as gpt3.5turbo (OpenAI, 2023), fail to show significant understanding of the task and yield relatively low performance after exhaustive experiment.
Notably, recent developments in Pre-Trained Language Models (PTLMs) on text classification tasks (Howard and Ruder, 2018) have empowered us to build our system the other way round. After the compilation of paired ADUs of propositions and locutions nodes embedded in a meticulously designed textual prompt, we fine-tuned our PTLMs on the reconstituted dataset as that of a traditional text classification task (Wang et al., 2023c; Peng et al., 2024; Yan et al., 2024). Using this method, we were able to achieve relatively high accuracy in the identification of illocutionary relations. The classification results of Task B were then used as textual information to assist the identification of propositional relations.
An extensive ablation study was also conducted to test the effectiveness and generalizability of our proposed system. A maximal F1 score of 78.90% and precision of 82.35% on Task B was achieved using a fine-tuned DeBERTa-v3-base model (Howard and Ruder, 2018). It is also noted that DeBERTav3-large underperforms its base version, with a precision difference of -0.2%. The proposed explanation is that the model already converges on the given dataset, provided the base version parameters. Several other PTLMs, including RoBERTa (Liu et al., 2019) and ALBERT (Lan et al., 2020) are also trialed using identical prompt design, which yield results inferior to DeBERTa-v3.
However, fine-tuned PTLMs converges inconsistently for Task A, with a recall of only 33.79%. We suspect that besides text from adjacent propositions and locutions, the system will need more in-context information (e.g., a dialogue 2-3 nodes away) to assist the process of relation identification according to recent works on reasoning under contexts (Dong et al., 2024; Zhang et al., 2024; Li et al., 2024).
As such, our proposed system provides valuable insight for dialogical argument mining using PTLMs on a IAT layout, and future works should be more focused on the revamp of methodology in in-context training information extraction. Our code and results are publicaly available at Arwenwutietie/DialAM-2024

heading: 2 Problem Definition
text: In this section, we would introduce the dataset format and elaborate on the formal definition of the shared task in DialAM-2024.

heading: 2.1 Dataset Description
text: In the DialAM-2024 dataset, all input texts are categorized into two primary types: locutions (L-nodes) and propositions (I-nodes). Locutions represent the original sentence segments within a complete dialogue, typically featuring speakers and timestamps. Conversely, propositions are reconstructed locutions, where linguistic elements such as anaphora, pronouns, and deixis have been resolved. These two text types are then structured into a navigable graph based on IAT, with corresponding L-nodes and I-nodes connected by three distinct relation types: (i) relations between locutions in a dialogue, known as transitions (TA-nodes); (ii) relations between propositions and locutions (YA-nodes); and (iii) illocutionary connections that link locutions with their semantic content (S-nodes).
We use QT30 corpus (Hautli-Janisz et al., 2022) as our dataset. QT30 is a collection of 30 episodes of Question Time aired between June 2020 and November 2021, with a total of more than 29 hours of transcribed broadcast material and comprising 19,842 locutions by more than 400 participants. The QT30 dataset contains 10,818 propositional relations that include Default Inferences, Default Conflicts, and Default Rephrases, and 32,303 illocutionary relations divided into Asserting, Agreeing, Arguing, Disagreeing, Restating, Questioning, and Default Illocuting.

heading: 2.2 Task Definition
text: The DialAM-2024 challenge comprises two distinct sub-tasks. Task A aims to detect the argumentative relations that exist between the propositions identified and segmented within the argumentative dialogue. More specifically, the objective is to use two connected I-nodes to predict the S-nodes between them. Task B, on the other hand, seeks to
identify the illocutionary relations that exist between the locutions uttered in the dialogue and the argumentative propositions associated with them. In other words, given a set of locutions (L-nodes) and propositions (I-nodes), the goal is to uncover the Illocutionary connections (YA-nodes) that link them.
To allow us to establish a clear and formal framework for analyzing the relationships. Formally, let us denote two coherent locutions as L1 and L2, their corresponding propositions as I1 and I2, the intermediate TA-nodes between L1 and L2 as T , the YA-nodes connecting L1 and I1 as YLI1 , the YA-nodes connecting L2 and I2 as YLI2 , the intermediary S-nodes between I1 and I2 as S, and the YA-nodes connecting T and S as YTS . We denote LLMs as F and the curated prompt as P1, P2 respectively for Task A and Task B. By these notations, the Task A and Task B could be reformatted formally as:
Task A: S = max i F (Si|I1, I2, P1);
Task B: YLI1 = max i F (YLIi |I1, L1, P2),
where Si and YLI1 denote the output of PTLMs.

heading: 3 System Overview
text: In this section, we will introduce our proposed system. Our method conducts sequential inferences where we predict YLI1 , YLI2 and S in the first stage, then infer YTS with the predicted S in the previous stage.

heading: 3.1 Prompt Design
text: With the rapid advancement exhibited in prompt engineering technique (Chang et al., 2024; Qiao et al., 2023; Xu et al., 2024) it has been pointed out that prompting makes better use of the pre-trained data of PTLMs, allowing the model to perform better on fewer training examples, which can be helpful when classifying classes with smaller examples in this task. Being aware of this, since this text classification task is highly specified and targeted, we meticulously curated descriptive prompting for both sub-tasks. The prompt is then aggregated with given texts as the inputs for large model. Predefined special tokens like [SEP], [CLS] and [EOS] are also added to the final input texts to assist the model to understand the relationship between the different parts of the input. Totally, three different prompts have been used for Task A and B: P1 (prompt used to predict YLI1 and YLI2), P2 (prompt used to predict S) and P3 (prompt used to predict YTS).

heading: 3.2 Sequential inference and model training
text: Recently, decomposing complex problems into several simple one has become a fashion in LLM reasoning field (Bueno et al., 2024; Besta et al., 2024). Following this trend, in this project, the training of PTLMs is divided into three sequential stages, as shown in figure 2.

heading: 3.2.1 Stage 1: Direct Illocutionary Relation Detection (Task B.1)
text: In Stage 1, we instruct the model to predict YLI1 and YLI2 separately, since the illocutionary rela-
tions between L-nodes (L1 and L2) and I-nodes (I1 and I2) is more intuitive and requires less information to classify. The raw textual prompt used is (′[CLS]′+P1+′ [SEP ]′+L1+′ [SEP ]′+I1) and (′[CLS]′ + P1 +′ [SEP ]′ + L2 +′ [SEP ]′ + I2).

heading: 3.2.2 Stage 2: Propositional Relation Detection (Task A)
text: Then, in Stage 2 we subsequently classify S-nodes with textual prompt (′[CLS]′ + P2 +′ [SEP ]′ + I1 + ′ [SEP ]′ + I2).

heading: 3.2.3 Stage 3: Indirect Illocutionary Relation Detection (Task B.2)
text: Finally, motivated by our observation that S and YTS are highly related, we incorporate the information yield through the previous two stages. Specifically, we leverage L1, L2 and the already predicted S for the prediction of YTS . The prompt we used is (′[CLS]′ +P3 +′ [SEP ]′ +L1 +′ [SEP ]′ + S +′ [SEP ]′ + L2 +′ [SEP ]′).

heading: 3.2.4 Training Objective
text: All models are trained with cross-entropy loss. Denote each input as xi, its token length as |xi|. Our models are denoted by p, and thus p(xi) represents the prediction made by the corresponding node, with q(xi) as its true label.
L(xi, q) = − |x|∑
i=1
p(xi) log(q(xi)) (1)

heading: 4 Experimental Setup
text: We followed a standard approach to partition our input data into training and validation sets. Please refer to Appendix C for more details.

heading: 5 Results and Analysis
text: In this section, we demonstrate our experiment results and conduct analysis on the issue we encountered through the experiments.
Our overall result is shown in Table 1. From the data we can observe that both DeBERTa-base and DeBERTa-large can achieve a relatively high accuracy on the prediction of YLI1+YLI2 , YTS and S. However, ChatGPT’s results were clearly not satisfactory, and it achieved the lowest accuracy rate on all 3 tasks. The reason could be that this text classification task is highly specialized and targeted where related resources rarely occur in ChatGPT’s training data. Consequently, ChatGPT would fall short in relevant reasoning tasks. In the classification of YLI1+YLI2 , we realize that the most numerous type in YLI1+YLI2 , Asserting, accounts for 90% of the total number of YLI1+YLI2 . We suspect that this may affect the final performance of the model, making it more inclined to split a new YLI1orYLI2node into the Asserting class. Based on this, we tried to reduce the number of Asserting classes in the training set to train a more comprehensive model. However, the final results demonstrated that this actually led to a decrease in the overall accuracy. This implies that the model is scarcely affected by the imbalance of the dataset.
Further experiments indicate that the accuracy of S-node classification is greatly affected by the size of the training set. According to our observation, when 60% of the data is sampled for training, the accuracy on the test set reaches the highest (65.73%), and when all data is used for training, the accuracy decreases to 56.76%. We suspect that this may be due to model’s overfitting to the training data.

heading: 6 Conclusion
text: In this paper, we present our system for the DialAM-2024 dialogical argument mining task, focusing on the identification of propositional and illocutionary relations within dialogues. By leveraging the IAT framework, we developed a methodology that integrates human-defined prompts to
stimulate PTLMs’ reasoning. Our approach features commendable results in the identification of illocutionary relations with concise preprocessing procedures, as evidenced by our high F1 score and precision in Task B. Despite the notable success in Task B, our system encountered challenges in Task A, particularly in achieving consistent recall rates. This indicates that additional context beyond adjacent propositions and locutions may be necessary for enhancing the identification of argumentative relations. Our findings contribute valuable insights into the application of PTLMs in dialogical argument mining. The results underscore the importance of designing effective prompts and highlight the need for ongoing methodological advancements to fully harness the capabilities of PTLMs in complex argumentation analysis tasks.
Ethics Consideration
The authors believe that this paper does not yield additional ethics concerns. All models and datasets accessed are freely accessible for research purposes.

heading: B Prompt design
text: P1="Illocutionary relations include 0:Asserting, 1:Pure Questioning, 2:Challenging, 3:Assertive Questioning, 4:Rhetorical Questioning, 5:Agreeing, 6:Default Illocuting, 7:Arguing, 8:Restating, 9:Disagreeing.The illocutionary relation between the two sentences is [mask].".
P2="Illocutionary relations include 0:Default Inference, 1:Default Rephrase, 2:Default Conflict.The illocutionary relation between the two sentences is [mask].".
P3="Illocutionary relations include 0:Asserting, 1:Pure Questioning, 2:Challenging, 3:Assertive Questioning, 4:Rhetorical Questioning", 5:"Agreeing", 6:"Default Illocuting", 7:"Arguing", 8:"Restating", 9:"Disagreeing".The illocutionary relation between the two sentences is [mask].".

heading: C Experiment Setup
text: We allocated 80% of the data to the training set, while the remaining 20% was assigned to the validation set. Prior to training, the datasets were tokenized and then fed into language models for fine-tuning. The learning rate was set to 2e-5, and the model underwent training for 2 epochs. To update the model’s parameters, we employed the AdamW optimizer.
During the evaluation phase, we assessed the model’s performance on the validation using accuracy as the metric. This metric takes the model’s predictions and the ground-truth label as input and returns the portion of the correct predications. Every epoch, we printed out the achieved accuracy. To ensure optimal model performance, we conducted experiments with various input sizes and epochs, aiming to strike a balance between underfitting and overfitting.
To support our computations, we leveraged a single NVIDIA RTX A6000 card as our computational infrastructure. The best checkpoint, determined by our experiments, was utilized to generate the submitted maps.

heading: Future Work
text: As such, our proposed system provides valuable insight for dialogical argument mining using PTLMs on a IAT layout, and future works should be more focused on the revamp of methodology in in-context training information extraction. Our code and results are publicaly available at Arwenwutietie/DialAM-2024