Discourse Structure-Aware Prefix for Generation-Based End-to-End Argumentation Mining (2024)

Abstract: End-to-end argumentation mining (AM) aims to extract the argumentation structure including argumentation components and their argumentation relations from text. Recent developments in end-to-end AM models have demonstrated significant progress by redefining the AM task as a sequence generation task, exhibiting simplicity and competitive performance. Nevertheless, these models overlook the integration of supplementary discourse structure information, a crucial factor for comprehending argumentation structures, resulting in suboptimal outcomes. In this study, we propose the DENIM framework, which generates discourse structure-aware prefixes for each layer of the generation model. These prefixes imbue the generation-based AM model with discourse structures, thereby augmenting the overall generation process. Moreover, we introduce a multi-task prompt coupled with a threestep decoding strategy, aiming to optimize the efficiency and effectiveness of argumentation structure decoding. Extensive experiments and analyses on two benchmark datasets show that DENIM achieves state-of-the-art performances on two AM benchmarks.

heading: 1 Introduction
text: Argumentation mining (AM) has recently received much research attention (Palau and Moens, 2009; Stede et al., 2019; Lawrence and Reed, 2020), which aims to analyze and understand argumentation texts to obtain argumentation structure.
Generally, AM generally comprises three subtasks following (Morio et al., 2022): 1) argumentation component segmentation (ACS) detects the boundaries of token-level argumentative segments, which are known as ACs; 2) argumentation component type classification (ACTC) classifies the
* Equal Contribution. † Min Yang and Ruifeng Xu are corresponding authors.
segment-level ACs into the categories (i.e., Claim and Premise); 3) argumentation relation classification (ARC) further classifies the AR types (i.e., No-Relation, Support and Attack) between AC pair. The end-to-end AM task is highly challenging due to the complexity of simultaneously addressing these three subtasks within a unified framework.
Early studies focus on only a subset of the three subtasks (Niculae et al., 2017; Bao et al., 2021a) in AM. Recently, some studies have formulated the end-to-end AM as a dependency parsing task (Eger et al., 2017; Ye and Teufel, 2021). However, dependency parsing requires a complex pre-processing and post-processing process to match the argumentation structure with the elaborately designed dependency graph. Morio et al. (2022) tackled AM by combining sequence labeling and multi-class classification tasks within a multi-task learning framework. Inspired by the success of the generative methods, a generation-based end-to-end AM model (Bao et al., 2022) is proposed to transform the AM as a unified generation task.
Compared to traditional classification-based methods (Ye and Teufel, 2021; Morio et al., 2022), recently proposed generative AM models (Bao et al., 2022) have demonstrated simplicity in task decoding and competitive performance. However, existing generative approaches primarily focus on problem reformulation and, unlike traditional classification-based methods, do not incorporate additional discourse structure information, which has been proven effective for argument mining in prior works (Accuosto and Saggion, 2019, 2020). As shown in Figure 1, the discourse structure graph derived from the input text summarizes the semantic structure of the text, where the elementary discourse units (EDUs) are nodes and the discourse relations (e.g., Contrast) between EDUs form edges. In this graph, many nodes and edges
11597

heading: EDU1: Despite the fact
text: share strong similarities with the argumentation structure. For example, the AC3 “ They introduce new products” can be mapped to the EDU5, and its relative AC can be found using edge “Contrast”. Hence, we argue that the discourse structure graph could provide important clues for generative-based AM models to figure out the argumentation structure, potentially enhancing overall performance and improving generalizability.
To address the aforementioned issue, we propose DENIM (Discourse structure-aware prEfix geNeration-based argumentatIon Mining), a generation-based end-to-end AM model that incorporates discourse structure information into prefix (Li and Liang, 2021) to guide the generation-based AM models. Specifically, we employ an additional discourse structure encoder to convert the input discourse structure graph into dense vectors. Then, these vectors will be disassembled and allocated to each Transformer layer within generation-based AM models as prefixes. These generated prefixes are converted into supplementary key and value matrices, exerting influence on the attention calculation process and guiding the generation mechanism.
Furthermore, we design a multi-task prompt using a fixed template and a three-step decoding strategy tailored for AM. DENIM initially produces spans of all ACs for ACS, subsequently classifies their types for ACTC, and ultimately identifies the
relations between AC pairs for ARC. The computation process in each step adheres to a fixed format and is parallelized on GPUs. Compared to the previous generation-based AM model (Bao et al., 2022) using an autoregressive decoding strategy, our method demonstrates enhanced efficiency through task-specific decoding and improved controllability with a fixed prompt template.
Our contributions can be summarized as follows. (1) We propose a discourse structure-aware prefix to encode discourse structure information for the generation-based end-to-end AM model. (2) We design a multi-task prompt with a three-step decoding strategy for AM. This approach is more effective and efficient than the traditional autoregressive decoding strategy. (3) We conduct extensive experiments on two AM benchmark datasets. The experimental results present that our method significantly outperforms the strong baselines.

heading: 2 Methodology
text: DENIM uses BART (Lewis et al., 2019) as the base model, and augments it with discourse structureaware prefix and multi-task prompt with a threestep decoding strategy, as depicted in Figure 1. To generate the discourse structure-aware prefix, we first employ a pre-trained discourse parser to extract the discourse structure graph of the input text (Section 2.2). Then, the graph is transformed into dense vectors through a discourse structure encoder.
Subsequently, these dense vectors will be disassembled and distributed to each layer of the BART so the generation is guided by the discourse structure information (Section 2.3). Lastly, we introduce a multi-task prompt with a three-step decoding strategy to alleviate the time-consuming and instability of sequence decoding for effectively and efficiently generating the argumentation structure (Section 2.4).

heading: 2.1 Task Definition
text: Following previous works (Morio et al., 2022), for the end-to-end AM, the input is a piece of argumentation text W = {w1, w2, . . . , wn} consisting of n tokens. The first goal is to extract a set of ACs A = {ai|ai = (si, ei)}mi for ACS, where ai is the i-th AC, si and ei denote its start and end indexes respectively, and m denotes the number of ACs in the text. Next, the type ci (i.e., Claim and Premise) of each extracted AC should be predicted for ACTC. Finally, the argumentation relation rel(i,j) (i.e., Norelevant, Support and Attack) between each AC pair (ai, aj) should be identified for ARC.

heading: 2.2 Discourse Structure Parsing
text: The first step of our method is to prepare the discourse structure graph (DSG) of the input text. We consider an external discourse parser called DMRST (Liu et al., 2021) to automatically parse the input text. As illustrated by Figure 1, the discourse parser encodes the input text into a DSG G =< V,E >, where each node vi ∈ V represents an EDU. The edge set, E ⊆ V ×R×V , comprises edges ei,j ∈ R|R| ⊆ E, each indicating a distribution over all discourse relations between two EDUs (vi, vj). Here, we consider three types of common discourse relation types R = {Expansion,Contingency, Comparison} like (Pu et al., 2023) 1.
Note that we utilize the distributions over all discourse relation labels between two EDUs to enhance the heterogeneous graph, rather than restricting to the 1-best result (i.e., the relation with the highest predicted probability) inspired by (Pu and Sima’an, 2022). This strategy offers two advantages. Firstly, it mitigates error propagation from the external discourse parser by using relation distributions. Secondly, as suggested by Yung et al. (2022), multiple intrinsic relations can coexist simultaneously between EDUs. Representing these
1Further details are presented in Appendix A.7
relations as a distribution rather than the 1-best result provides a more nuanced and accurate discourse structure. Thus, we argue that the logit output from the parser is more informative. It provides not just the n-best results but also captures the remaining uncertainty of predictions.
We validate the effectiveness of this strategy and our model’s sensitivity to discourse structure information through experiments detailed in Appendix A.5.

heading: 2.3 Discourse Structure-Aware Prefix Generation
text: Next, to enable the discourse structure information to guide the generation-based AM model, we use the prefix (Li and Liang, 2021) as the bridge between them and inject the discourse structure information into the prefix. We employ a discourse structure encoder comprising a BARTEncoder and a Relational Graph Convolutional Network (RGCN) (Schlichtkrull et al., 2018) to model the discourse structure graph. The BARTEncoder first learns the initialized node representation and the RGCN models the graph structure by information propagation. Specifically, we utilize the BART-Encoder to encode the argumentative text W and obtain the hidden states HW = BARTEncoder(W ). Then, for i-th node in the DSG G, we mean-pool its context corresponding representations from HW to obtain the initialized node feature hi.
To let the nodes in DSG interact with each other and model the discourse structure information, we perform graph-based information propagation (Gilmer et al., 2017) to update node representations. Since DSG is a heterogeneous graph containing multiple types of edges, we utilize the relation-specific node-to-node information propagation mechanism in RGCN to model the DSG. Specifically, given a node v ∈ G at the c-th RGCN layer, the information propagation and aggregation operation is defined as follows:
hc+1v = ReLU(h c v +
∑
r∈R
∑
u∈Nr(v) Êv,r,uW
c rh c u + b c r) (1)
where Nr(v) denotes the neighbors for node v connected with the edge of type r, ReLU is the ReLU activation function, W cr and b c r are the trainable parameters. Êv,r,u = Ev,r,u∑
o Ev,r,o+1 is the normal-
ized edge weight between the node v and u with relation r. Finally, we select the representation of
all nodes in the last layer as the updated node representations, which aggregate discourse structure information from a certain heterogeneous graph.
After that, we introduce an attention layer (Vaswani et al., 2017) and z learnable vectors as queries, where z is a hyperparameter determining the length of the used prefixes. These queries interact with the updated node representations, serving as both keys and values in the attention mechanism. The output of the attention layer is a set of dense vectors, denoted as P , which effectively condense the discourse structure information from the node representations.
We then transform these vectors into prefixes, similar to (Li and Liang, 2021), to be incorporated into our generation-based AM model. Concretely, this transformation involves partitioning P into L pieces, corresponding to the number of layers in the generation-based AM model, i.e., P = {P 1, . . . , PL}. For the i-th layer, the prefix is further divided into two matrices representing additional keys and values matrices: P i = {Ki, V i}, where Ki and V i are the addition key and value matrices, and they can be further written as Ki = {ki1, . . . , kiz} and V i = {vi1, . . . , viz}. ki∗ and vi∗ are vectors with the same hidden dimension in the i-th Transformer layer. These additional key and value matrices will be concatenated with the original key and value matrices in the attention block. Consequently, when calculating dot-product attention, the query at each position will be influenced by these discourse structure-aware prefixes.
It is worth noting that we assign different keyvalue pairs for different layers, offering two benefits. 1) the layer-wise queries and keys can exert strong control. 2) Different layers may need to embed varied information (Clark et al., 2019; Rogers et al., 2021). In addition, DENIM generates a distinct set of prefixes when the input text varies. The variation reflects the different discourse structure graph’s presentation. This differs from the prefix tuning technique (Li and Liang, 2021) uses a fixed set of prefixes for all input instances.
We can integrate prefixes into the encoder selfattention blocks, decoder cross-attention blocks, or decoder self-attention blocks of our generationbased AM model. Based on our preliminary experiments in Appendix A.3, we observe that using prefixes in the self-attention blocks of the encoder and decoder works best in DENIM.
ACS Subtask
(𝒔𝒑𝒂𝒏𝟏, 𝒕𝒚𝒑𝒆𝟏, 𝒔𝒑𝒂𝒏𝟐, 𝒕𝒚𝒑𝒆𝟐, 𝒓𝒆𝒍𝟏𝟐)
(a) Autoregressive Decoding
(b) Our Multi-Task Prompt with Three-Step Decoding
(𝒔𝒑𝒂𝒏𝟏, 𝒕𝒚𝒑𝒆𝟏, 𝒔𝒑𝒂𝒏𝟑, 𝒕𝒚𝒑𝒆𝟑, 𝒓𝒆𝒍𝟏𝟑)
𝒔𝒑𝒂𝒏𝟏 𝒔𝒑𝒂𝒏𝟐 𝒔𝒑𝒂𝒏𝟑
ACTC Subtask
𝒕𝒚𝒑𝒆𝟏 𝒕𝒚𝒑𝒆𝟐 𝒕𝒚𝒑𝒆𝟑
ARC Subtask
𝒓𝒆𝒍𝟏𝟐 𝒓𝒆𝒍𝟏𝟑 𝒓𝒆𝒍𝟐𝟑
Figure 2: Comparison of decoding process between autoregressive decoding and our three-step decoding strategy. Top panel: the autoregressive decoding decodes the target sequence by token-by-token generation. Bottom panel: our three-step decoding employs a fixed prompt template and task-by-task decoding for AM.

heading: 2.4 Multi-Task Prompt with Three-Step Decoding
text: We design a more effective and efficient multitask prompt with a three-step decoding strategy to extract the argumentation structure for AM, compared with the previous generationbased AM model (Bao et al., 2022) using an autoregressive decoding strategy. Specifically, as shown in the top panel of Figure 2, the previous generation-based AM model formulates AM as a sequence generation task and the target sequence comprises several tuples. Each tuple represents an AC pair and their relation in the format [span1, type1, span2, type2, rel1,2] where spani denotes the start/end indexes of i-th AC, typei is type (i.e., Claim and Premise) of i-th AC and reli,j denote the argumentation relation (i.e., Support and Attack) between the AC pair (i, j). They apply an autoregressive decoding strategy to decode the target sequence by token-by-token generation. It not only is time-consuming but also may cause invalid predictions since the sequence generation is not fully controllable (Bao et al., 2022).
To mitigate the issues of time-consuming and uncontrollable token-by-token generation of autoregressive decoding (Bao et al., 2022), as discussed in Section 1, we introduce a multi-task prompt with a three-step decoding for AM, as shown in Figure 2. In particular, given an argumentative text W , we assume it has at most m ACs 2 and m× (m− 1)/2
2We heuristically set m to the maximum number of ACs in the train set.
AC pairs, the prompt template can be defined as:
T = Concat(TACS , TACTC , TARC) TACS = [S1] . . . [Sm] TACTC = [AC1] . . . [ACm] TARC = [P(1,2)] . . . [P(m−1,m)]
Where [Si], [ACi] and [P(i,j)] placeholders 3 denote the span, type of AC ai and relation between the AC pair (ai, aj) for ACS, ACTC and ARC subtasks, respectively. We use a fixed number of template tokens for all instances though different instances have different numbers of ACs. These redundant placeholders for each instance can be recognized as invalid tokens during training and inference. We feed the prompt template into the BART-Decoder to decode the argumentation structure for AM.
Training The training objective of DENIM is to predict gold labels based on the placeholders in the prompt template for AM. Specifically, DENIM is expected to derive the i-th AC span (si, ei) from the input text W using the placeholder [Si] for the ACS subtask. For that, we employ the output hidden state hSi of the [Si] to get the span selector θi = {ψsi , ψei}, where ψsi = wstarthSi , ψei = wendh S i , and wstart and wend are learnable parameters. Then, the start/end indexes (si, ei) of the i-th AC can be obtained by calculating the distribution of each token in the input context W :
si = argmax 0≤k≤n
softmax(ψsiH W )
ei = argmax 0≤k≤n
softmax(ψeiH W )
(2)
For ACTC, DENIM predicts the type of the i-th AC using the output hidden state of the placeholder [ACi]. To align the prediction of the placeholder [ACi] with the i-th AC span, we add the context representation hai = 1 ei−si+1 ∑ei k=si
HWk of the gold AC ai into the embedding of the [ACi] before feeding it into the BART-Decoder. Similarly, we add the sum of context representations hai,aj = 1 2(hai + haj ) of the gold AC pair (ai, aj) and the embedding of their gold type into the embedding of the placeholder [P(i,j)] and feed it into the BARTDecoder. DENIM predicts the relation between AC pair (ai, aj) using the output hidden states of the placeholder [P(i,j)]. The loss functions of all subtasks are cross-entropy losses.
3The placeholders are also called virtual words and implemented by using different specific tokens similar to eos token < s > in the BART vocabulary.
Take Figure 1 as an example, DENIM is expected to enable [S1] to predict the AC span (“advertisements can be exaggerated”), then use [AC1] to predict the type (“Premise”) of the AC a1, and use [P1,3] to predict the relation (“Attack”) between AC pair (a1, a3). If there is no predicted AC for one placeholder [Si], the model should predict the last position of the input W (i.e., specific token “
”) and the [ACi] and [Pi,j ] are invalid.
Inference During inference, our prompt template utilizes a three-step decoding strategy, differing from the token-by-token autoregressive decoding in the prior generation model (Lewis et al., 2019; Bao et al., 2022). Initially, DENIM inputs all placeholders [S∗] into the decoder to derive all AC spans. This step sets the foundation for subsequent decoding. In the second step, placeholders [AC∗] with the context representation of the predicted AC spans are fed into the decoder to predict the type of ACs, as in the training phase. Finally, DENIM calculates the relation between all AC pairs using the placeholder [P(∗)] with the sum of the context representation of the predicted AC pair and the embedding of the AC type. Notably, the decoding of the placeholders in the later subtask depends only on the result of placeholders in the former subtask. Thus, our prompt template only needs to be fed into the decoder three times, which saves time compared to autoregressive decoding. Furthermore, the prompt template serves as a strong control signal and defines the expected output format to ensure the output is valid.
Efficiency Considerations DENIM comprises three major components including a BART, an RGCN and a prompt template. The RGCN is required to model the discourse structure graph, with a time complexity of O(|R||V |2) that may cause efficiency considerations. In practice, this issue is minor for our experiments on the two datasets (AAEC and Abstract) and real scenarios, as the number 4 of EDUs in argumentation texts is consistently small in scale. Additionally, when incorporating the prefix into the BART-Encoder, the input context needs to be processed twice by the BARTEncoder. The first time is to construct the prefix and the second is to model the context representation of the input for the decoder. Its time cost is acceptable, as demonstrated in Appendix A.6.
4The average number of EDUs for the samples in the two datasets is 6 and 17, respectively.

heading: 3 Experimental Setup
text: Datasets To evaluate the effectiveness of our DENIM model, we conduct extensive experiments on two widely used AM datasets, named: AAEC (Stab and Gurevych, 2017) and AbstRCT (Mayer et al., 2020).
Evaluation Metrics We employ the same evaluation metrics as the previous works (Morio et al., 2022; Cheng et al., 2022; Guo et al., 2023), including F1 score and macro averaged score (denoted as Macro). We calculate F1 scores for determining the AC span for ACS. We adopt the F1 and macro averaged score for ACTC and for determining the relation type (not including No-Relevant) for ARC. We also introduce the Link score (Kuribayashi et al., 2019), used to measure F1 scores for identifying the existence of relations regardless of their types.
Baselines We compare DENIM with the following strong baseline models. For the AAEC dataset, we compare our model with five strong baselines, including BiPAM (Ye and Teufel, 2021), BiPAMsyn (Ye and Teufel, 2021), BART-B (Yan et al., 2021), GMAM (Bao et al., 2022) and ST (Morio et al., 2022). For the AbstRCT dataset, we compare our model with fhree strong baselines, which are BART-B (Yan et al., 2021), GMAM (Bao et al., 2022) and ST (Morio et al., 2022).
Implementation Details DENIM is implemented in PyTorch on an NVIDIA TESLA A100PCIE-40GB and employs the BART base5. Our model is optimized using AdaW (Loshchilov and Hutter, 2017) with the learning rates of 3e-5 and weight decay of 1e-5 on both AbstRCT and AAEC datasets. For both datasets, we set the batch size to 4 and adopt dropout (Srivastava et al., 2014) with a dropout rate of 0.1 to avoid overfitting. We set the layer number L of RGCN to 1 because of its best performance. All experiments are performed five times with different random seeds, and the evaluation scores are averaged. Our code is available at https://github.com/syiswell/DENIM.

heading: 4 Experimental Results

heading: 4.1 Overall Performance
text: We report the overall performance of our proposed framework and baseline methods in Table 1. Our
5We implement BART using huggingface toolkit: https://huggingface.co/
method achieves the best performance on both AAEC and AbstRCT datasets. For example, on AAEC, DENIM exceeds the current state-of-theart (SOTA) method ST and obtains about 6.74% higher Macro score on the ACTC subtask. On AbstRCT, DENIM outperforms ST by 7.23% in terms of F1 score on the ACS subtask. The experimental results verify the superiority of our method for AM.
We also observe that the generation-based models (i.e., BART-B and GMAM) surpass the dependency parsing-based methods (i.e., BiPAM and BiPAM-syn), indicating that formulating the endto-end AM task as a generation task might be more effective than as a dependency parsing task. While BART-B and GMAM underperform the SOTA method ST, they typically offer a simpler decoding paradigm for AM. Based on the generation-based method, our DENIM performs better than all strong baselines by integrating a discourse structure-aware prefix and a multi-task prompt.

heading: 4.2 Ablation Study
text: To analyze the impact of different components in our proposed DENIM method, we conduct ablation studies in terms of removing RGCN (w/o RGCN), removing discourse structure-aware prefix (w/o Prefix), and removing multi-task prompt (w/o Prompt), respectively. Note that w/o Prompt means removing the prompt and transforming the taskspecific decoding into token-by-token decoding while the target sequence is similar to the prompt format. The results are reported in Table 2.
We can observe that w/o RGCN degrades the performance, verifying that the discourse relations are beneficial for AM. w/o Prefix leads to further performance drops, demonstrating that discourse structure-aware prefixes can provide crucial clues to figure out the argumentation structure. Note that w/o Prefix outperforms previous generation-based methods (GMAM and BART-B) and SOTA method ST, verifying that our multi-task prompt with threestep decoding is more effective than them. Removing the multi-task prompt (w/o Prompt) leads to a noticeable drop in performance due to the absence of an efficient output format and strong control signals from the prompt.

heading: 4.3 Adaptability Experiment
text: Inspired by the recent success of Large Language Models (LLMs) (Min et al., 2023), we conduct additional experiments on two prominent LLMs, namely ChatGPT-3.5-Turbo and LLAMA2 (Tou-
vron et al., 2023). We employ the natural language prompt approach like (Madaan et al., 2022; Li et al., 2023) and few-shot in-context learning (here we employ 3-shot due to the limitation of input length) in ChatGPT-3.5-Turbo and LLAMA2 for AM. Figure 4 displays the format of the natural language prompt for AM. To evaluate the versatility of our method, we adapt it to fine-tune LLAMA2 using LORA (Hu et al., 2021), denoted by LLAMA2DENIM. Furthermore, we fine-tune LLAMA2 with the natural language prompt approach (denoted by LLAMA2-FT) and LLAMA2-DENIM without prefix (w/o Prefix) as baselines. Each finetuning method uses a batch size of 1. Unfortunately, due to computational resource limitations, we are unable to perform fine-tuning experiments using LLAMA2 on the AbstRCT dataset. The results are presented in Table 3.
We observe that few-shot-based methods (i.e., LLAMA2 and ChatGPT-3.5-Turbo) significantly underperform fine-tuning methods (i.e., DENIM, LLAMA2-DENIM and LLAMA2-FT) on AAEC. Among these fine-tuning approaches, our DENIM framework with LLAMA2 (i.e., LLAMA2DENIM) outperforms the w/o Prefix as well as the LLAMA-FT which is fine-tuned using natural language prompt, validating the effectiveness
of our methods. In addition, LLAMA-DENIM performs worse than DENIM using BART as the base model. This discrepancy can be attributed to LLAMA-DENIM’s large number of parameters (including trainable and non-trainable parameters) while the AM datasets have a small number of samples (see Data Statistics), which leads to severe overfitting.

heading: 4.4 Impact of Different Ways for Discourse
text: Information Incorporation
We compare different ways to incorporate discourse structure information into the generationbased AM model: 1) discourse structure-aware prompts for Encoder (denoted by DENIM-E). We append the output representations of RGCN to the input text embeddings as the prompt. 2) encoding concatenation (denoted by DENIM-C). We concatenate the output representations of RGCN with the output representation of the BART-Encoder and feed them together to the BART-Decoder. 3) discourse structure-aware prompts for Decoder (denoted by DENIM-D). We concatenate the output representations of RGCN to the embeddings of the multi-task prompt template as part of the prompts. We present the results in Table 4. We can observe that DENIM outperforms all variants. An interesting finding is that DENIM-E is worse than the model w/o Prefix in Table 2. This is because the heterogeneous nature between discourse structure graphs and natural language sentences of encoder input would be confusing for models. The DENIMC achieves better performance than DENIM-D and DENIM-E by concatenating the discourse structure information with high-level output representations of the encoder instead of natural language sentences and formatted prompts.

heading: 4.5 Impact of Different Prompt Template
text: We explore two multi-task prompt variants: 1). DENIM-I uses identical placeholders for each subtask to assess placeholder effects. 2). DENIMR swaps ACTC and ARC placeholders to examine the impact of task order bias from the left-toright generation paradigm of the BART. In addition, we introduce a two-step decoding strategy for our prompt (denoted by DENIM-T), where the ARC does not depend on the results of ACTC. During decoding, DENIM-T first decodes all AC spans and then predicts the type of AC and the relation between AC pairs simultaneously. We describe the prompt formats and decoding process for the three variants in detail in Appendix A.4.
The results of all variants of prompt templates on AAEC and AbstRCT are shown in Table 5. We observe that our prompt template gets better performance than DENIN-I, demonstrating different prompt tokens may contain specific semantics for different ACs and AC pairs. We also focus on DENIM-R making performance degradation compared to DENIM, confirming the negative effect caused by the order biases between tasks, which has been explored in previous work (Bao et al., 2022). Unsurprisingly, DENIM-T’s performance is slightly worse than DENIM’s due to placeholders
in ARC not being able to access the information of types from the placeholders in ACTC, and vice versa.

heading: 4.6 Case Study
text: We conduct a case study to explain the benefit of integrating discourse information into AM intuitively. We compare DENIM and w/o Prefix and demonstrate two exemplary cases in Figure 3 to show the difference in their results.
Example A illustrates a typical case in which the EDUs in the DSG provide hints for the model to accurately segment the boundaries of the AC span. Specifically, the argumentation text is divided into five EDUs, two of which have strong similarities with the ACs of the argumentation structure. Without the DSG information, w/o Prefix will segment AC insufficiently, while DENIM can correctly segment two ACs in a lengthy sentence.
Example B shows how DSG helps DENIM perform ARC. In the DSG, both EDU2 and EDU3 have an “Elaboration” relationship pointing to EDU1. Given the high similarity of EDU1, EDU2 and EDU3 to AC1, AC2 and AC3, respectively, DENIM accurately identifies two AC pairs (AC1, AC2) and (AC1, AC3) with a “Support” relation, while w/o Prefix struggles to capture the relationship between AC pairs.

heading: 5 Related Work
text: Argumentation Mining Argumentation mining aims to extract the argumentation structure from the argumentation text, which involves three subtasks (Lawrence and Reed, 2020; Bao et al., 2021b; Cheng et al., 2021; Sun et al., 2022; Guo et al., 2023; Chen et al., 2023). Early work (Persing and Ng, 2016) performed joint inference in an Integer Linear Programming (ILP) framework. With the success of deep learning, Eger et al. (2017) investigated neural techniques for AM and formalized the end-to-end AM task into multiple other tasks. Ye and Teufel (2021) extended the approach of (Eger et al., 2017) using biaffine dependency parsing and achieved promising performance. However, it requires tedious pre- and postprocessing. Bao et al. (2022) proposed a generative framework with a constrained pointer mechanism and a reconstructed positional encoding for end-to-end AM. Morio et al. (2022) proposed a Longformer (Beltagy et al., 2020) based model with biaffine functions for AM.
In addition, some works explore incorporating discourse information for AM. (Green, 2010) introduced a non-deep learning approach that leverages Rhetorical Structure Theory to analyze argument structures within biomedical corpora. (Stede et al., 2016) introduced two theories, including Rhetorical Structure Theory and Segmented Discourse Representation Theory, to annotate and analyze argumentative texts. (Accuosto and Saggion, 2019) proposed an LSTM-based transfer learning approach that uses contextual representations learned from discourse parsing tasks as input for argument mining models. (Accuosto and Saggion, 2020) employed the annotated discourse units and
relations as auxiliary information and proposed two transfer learning approaches (i.e., multi-task learning and sequential learning) with BiLSTM and CRF models for AM. (Chistova, 2023) proposed a deep dependency parsing model to explore the relationship between rhetorical and argument structures. Different from them, we explore how to incorporate this additional information into a generation-based end-to-end AM and enhance it with prefix and prompt learning.
Prefix tuning and prompt tuning Prompt tuning (Liu et al., 2023) has achieved desirable performance in the field of natural language processing (NLP) (Schick and Schütze, 2021; Wang et al., 2022; Li and Liang, 2021; Ma et al., 2022; Hsu et al., 2023). Prefix tuning (Li and Liang, 2021), a member of the prompt-based tuning family, can trigger the desired generation of PLMs by only optimizing small continuous prefix vectors. To the best of our knowledge, we are the first to explore the potential of prefix tuning with discourse information for AM and propose an effective and efficient multi-task prompt with a three-step decoding strategy.

heading: 6 Conclusion
text: In this paper, we designed discourse structureaware prefixes, which introduce discourse structure information to the generation-based AM model, thereby enhancing the generation. Additionally, we introduce a multi-task prompt complemented by a three-step decoding strategy, optimizing the efficiency and effectiveness of argumentation structure decoding. Extensive experiments and analyses on two benchmarks show that our method outperformed strong baselines significantly.

heading: Limitation
text: To point out future research direction for generation-based AM models, we performed error analysis on 100 cases where our DENIM made mistakes for ACS, ACTC and ARC subtasks. On the ACS subtask, we identified two common types of errors: (1) incorrect AC span due to short segment. DENIM has lower confidence in the segmentation of short ACs compared to longer ACs. In particular, if an AC is a short segment, it tends not to be predicted as an AC or will be incorrectly spliced with subsequent clauses as an AC. (2) error AC combined with a main clause. For example, "It is generally understood that", "Many studies have pointed out that", etc. The former is not part of AC, while the latter is part of AC. DENIM finds it difficult to distinguish these subtle semantic differences. In DSG, both such examples are segmented into EDUs, which cannot provide effective hints for DENIM at the discourse structure level. For ACTC, the most serious problem is the error extraction of AC span, resulting in incorrect AC classification. In addition, we discover that there is a type bias for different positions of ACs. For instance, DENIM tends to predict the first AC as “Claim” and the last AC as “Premise”. This is because DENIM overfits the type distribution of ACs at different positions using the prompt placeholders. Maybe we can adopt a debias approach to alleviate this issue. On the ARC task, we find that DENIM captures the connection between long-distance ACs so excessively that it generates additional AC pairs. In the argumentation structure, AC pairs with relationships should not be too far apart, or it won’t follow the argumentation structure habits of human texts. Therefore, we suggest that during ARC tasks, incorporating distance loss between ACs can guide the model to focus more on the connection between ACs with moderate distances, which could potentially address the issue.
Last, we discuss the prefix technique proposed, which utilizes discourse structure graphs generated by a pre-trained discourse parser. These graphs, while effective, are not infallible and may contain imperfections, leading to potential error propagation issues in DENIM. Although employing distributions over all discourse relation labels helps mitigate this problem, it is not a perfect solution. A prospective approach to address this is to treat AM and discourse parsing as concurrent tasks within a multi-task learning framework. We leave this
integrated approach as our future work.

heading: Acknowledgements
text: We thank the anonymous reviewers for their valuable suggestions to improve the quality of this work. This work was partially supported by the National Key Research and Development Program of China (2022YFF0902100), the National Natural Science Foundation of China (62176076), Natural Science Foundation of Guangdong (2023A1515012922), the Shenzhen Foundational Research Funding (JCYJ20220818102415032), the Major Key Project of PCL2021A06, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005).

heading: A Appendix

heading: A.1 Data Statistics
text: • AAEC (Stab and Gurevych, 2017): The
AAEC dataset comprises 420 essays with 1833 paragraphs. There are three types of
ACs (i.e., MajorClaim, Claim and Premise) and two types of ARs (i.e., Support and Attack). Each AC has at most one outgoing AR so the argumentation graph of the paragraph can be either directed trees or forests. We divide this dataset into a training set of 1464 ACs and a testing set of 369 ACs, and randomly choose 10% of the training set as the validation set, which is consistent with previous works (Morio et al., 2022).
• AbstRCT (Mayer et al., 2020): The AbstRCT includes abstracts of randomized controlled trials (RCTs) from the MEDLINE database. All ACs are classified into three types: MajorClaim, Claim and Evidence. The ARs have three types: Support, Attack and Partialattack. Each AC may have several outgoing ARs, thus the argumentation graph is of nontree structure. We used 350 training, 50 development, and 100 test texts following (Morio et al., 2022).
The detailed statistics of the AAEC and AbstRCT datasets are summarized in Table 6.

heading: A.2 Baselines
text: We compare our proposed model with the following baselines:
• BiPAM (Ye and Teufel, 2021): This method applies BERT as the base model and transforms AM as a biaffine dependency parsing task for end-to-end AM.
• BiPAM-syn (Ye and Teufel, 2021): The BiPAM model is enhanced by explicit syntactic information produced by the Stanford syntactic dependency parser.
• ST (Morio et al., 2022): This method proposes an end-to-end AM model based on Longformer (Beltagy et al., 2020) and the biaffine function.
• BART-B (Yan et al., 2021): The model converts all aspect-based sentiment analysis subtasks into a unified generative formulation, which is adapted for extracting argument structure by (Bao et al., 2022).
• GMAM (Bao et al., 2022): A generative endto-end AM model reformulates AM as a sequence generation task, enhanced by reconstructed positional encoding and constrained pointer mechanism. Noted the BART-B is the basic model of GMAM without augmentation mechanism.

heading: A.3 Additional Ablation Study
text: Here, we explore which blocks (i.e., self-attention blocks in the encoder, self-attention blocks in the decoder, or cross-attention blocks in the decoder) we should place prefixes into. We find that (1) employing prefixes consistently improves the performance of our methods. (2) Using the prefixes only in the self-attention blocks of the encoder outperforms using it in another block. (3) We obtained the best performance by using prefixes in the selfattention blocks of the encoder and decoder, instead of using prefixes in all blocks. This may be because reusing prefixes in the self-attention and
ACS Subtask
(𝒔𝒑𝒂𝒏𝟏, 𝒕𝒚𝒑𝒆𝟏, 𝒔𝒑𝒂𝒏𝟐, 𝒕𝒚𝒑𝒆𝟐, 𝒓𝒆𝒍𝟏𝟐)
(a) Autoregressive Decoding
(𝒔𝒑𝒂𝒏𝟏, 𝒕𝒚𝒑𝒆𝟏, 𝒔𝒑𝒂𝒏𝟑, 𝒕𝒚𝒑𝒆𝟑, 𝒓𝒆𝒍𝟏𝟑)
𝒔𝒑𝒂𝒏𝟏 𝒔𝒑𝒂𝒏𝟐 𝒔𝒑𝒂𝒏𝟑
ACTC Subtask
𝒕𝒚𝒑𝒆𝟏 𝒕𝒚𝒑𝒆𝟐 𝒕𝒚𝒑𝒆𝟑
ARC Subtask
𝒓𝒆𝒍𝟏𝟐 𝒓𝒆𝒍𝟏𝟑 𝒓𝒆𝒍𝟐𝟑

heading: ACTC Subtask ARC SubtaskACS Subtask
text: cross-attention blocks of the decoder introduces redundant information.
A.4 Impact of Different Prompt Template
We study two variants of our multi-task prompt: 1). prompt template with identical task placeholders (denoted by DENIM-I). To explore the effect of placeholders in the multi-task prompt, we employ the same placeholder for different ACs or AC pairs in each subtask. 2). prompt template with task order reverse (denoted by DENIM-R). To study the effect of task order biases in the left-to-right generation paradigm of the BART decoder for DENIM,
we consider reversing the generation order between ACTC and ARC. For that, we reverse the position of placeholders for ACTC and the position of placeholders for ARI. The format of the two prompt templates can be viewed in Table 8.
In addition, we introduce a two-step decoding strategy for our prompt (denoted by DENIM-T) as shown in the bottom panel of Figure 5, where the ARC does not depend on the results of ACTC compared with the three-step decoding. During decoding, DENIM first inputs all placeholders [S∗] into the decoder to derive all AC spans, which is the same as the first step in three-step decoding. Then, placeholders [AC∗] with the context representation of the predicted AC spans, and [P(∗)] with the sum of the context representation of the two AC spans of the predicted AC pair, are fed into the decoder together. The placeholders [AC∗] and [P(∗)] are employed to predict the type of ACs and the relation between AC pairs simultaneously.

heading: A.5 Discussion of Different Discourse Structure Graph Construction
text: We utilize the distributions over all discourse relation labels between two EDUs to strengthen the heterogeneous graph, rather than a limitation to the 1-best result (i.e., the relation with the highest predicted probability) inspired by (Pu and Sima’an, 2022). This approach offers two advantages. Firstly, it mitigates error propagation from the external discourse parser by using relation distributions. Since the parser is known to perform poorly on out-of-domain data (Atwell et al., 2022), and may hence propagate errors into the AM model. Secondly, as Yung et al. (2022) suggests, multiple intrinsic relations can exist simultaneously between EDUs. Representing these relations as a distribu-
tion rather than the 1-best result provides a more nuanced and accurate discourse structure. Thus, we argue that the logit output from a discourse parser is more informative. It provides not just the n-best results but also captures the remaining uncertainty associated with these predictions.
To assess the efficacy of the proposed strategy, we conduct a set of experiments, incorporating a different variant. The variant termed the 1-best strategy (BS for short), utilizes the 1-best result to connect two nodes. The results are presented in Figure 6. Compared with the 1-best strategy,
our method exhibits a consistent superiority across all metrics. This enhancement is attributed to our strategy’s dual focus: mitigating error propagation from the discourse parser by leveraging a distribution over all relations, and acknowledging the potential coexistence of multiple intrinsic relations between EDUs to construct a more nuanced and accurate discourse structure.
Further exploration into the sensitivity of our method to discourse information involved the implementation of two additional variants. The second variant termed the EDU contamination strategy (ECS), introduces random contamination of k% of EDUs by changing its start/end indexes. The third variant, known as the relation contamination strategy (RCS), involves random contamination of k% of relations between two EDUs by substituting them with alternate relations. Surprisingly, we observed only a marginal decline in the model’s performance with increasing k values, significantly outperforming the previous state-of-the-art (SOTA) standard. These findings underscore the pivotal role of discourse information for AM and demonstrate our method’s commendable robustness in handling discourse information.

heading: A.6 Computational Cost
text: We investigate the computational cost of baseline methods and our DENIM model in training and inference. For a fair comparison, all these models use the same batch size of 4 in training and 1 in inference. Table 9 shows the training time and inference time on the AAEC and AbstRCT.
DENIM is faster than or comparable to the SOTA model (i.e., ST) for training, but slightly slower than the base model for inference. For example, DENIM shows an average increase of 0.003s and 0.007s in inference time per sample in AAEC and AbstRCT, respectively, compared to ST, which is acceptable in practice. In addition, during inference, our DENIM is more efficient than previous generation-based baselines (i.e., BART-B and
GMAM) by a factor of 5-20 owing to the multi-task prompt with a three-step decoding, although additional overhead is added to the training process.
Comparing DENIM with w/o Prefix, the prefix introduces a small amount of extra time (almost 13.2 seconds on AAEC and 10.2 seconds on AbstRCT for one epoch) in both the training and inference phases (6.23 seconds on PE and 0.93 seconds CDCP for all instances). This additional cost is acceptable.

heading: A.7 Selection and Categorization of Discourse Relation
text: We notice that some discourse relations parsed by the DMRST parser may be irrelevant to the argument mining task, and removing these irrelevant
relationships may benefit the AM task. A similar operation can be seen in prior work (Pu et al., 2023). For that, we analyze the number of overlaps between EDUs and ACs under each discourse relationship in the training set, as shown in Table 10. We observed that for relations such as Comparison, Condition, Enablement, and MannerMeans, the number of overlaps between EDUs with these relations and ACs was minimal, while the non-overlapping instances were significantly higher. We believe that EDUs that share text fragments with ACs provide more accurate cues for AM. To minimize the impact of these EDUs with low relevance and their associated relations, we chose to exclude these relations (named irrelevant relations) (The top six in Table 10).
To validate whether the irrelevant relations would affect performance, we explore four variants: 1) we employ all discourse relations to construct the DSG (denoted by Full-DSG). 2) we exclude irrelevant discourse relations and use the remaining relations to construct the DSG. (denoted by Exc-DSG). 3) we limit the difference among discourse relations and set the weight of edges in the DSG to 1 (denoted by unlabeled-DSG). 4) we set the weight of the edges with irrelevant relations to 0 in the DSG, while the weight of the remaining edges is set to 1 (denoted by unlabeled-Exc-DSG). The experimental results, presented in Table 11,
indicate that excluding irrelevant discourse relations is beneficial. Our strategy suggests that future research could focus on more effectively eliminating noise in discourse relations and better aligning them with argumentative relations.
In addition, we refer to the definitions of (Carlson and Marcu, 2001) to obtain the semantics of each discourse relation, as shown in Figure 7. Then, inspired by (Pu et al., 2023), we manually group relations with similar semantics, such as Reason and Explanation, and present the grouping description in Figure 7. To validate the effectiveness of our grouping strategy, we conduct additional experiments, including applying the random grouping strategy to both Full-DSG and Exc-DSG variants (denoted by Full-Random-DSG and Exc-RandomDSG, respectively). The results of these comparisons, as shown in Table 11, demonstrate the rationale of our approach.
We recognize that the strategies we employed for the selection and grouping of discourse relations exhibit certain limitations. The specific grouping strategies may confuse the true features of categories, which merit further investigation in future research endeavors.

heading: Future Work
text: Our strategy suggests that future research could focus on more effectively eliminating noise in discourse relations and better aligning them with argumentative relations. In addition, we refer to the definitions of (Carlson and Marcu, 2001) to obtain the semantics of each discourse relation, as shown in Figure 7. Then, inspired by (Pu et al. , 2023), we manually group relations with similar semantics, such as Reason and Explanation, and present the grouping description in Figure 7. To validate the effectiveness of our grouping strategy, we conduct additional experiments, including applying the random grouping strategy to both Full-DSG and Exc-DSG variants (denoted by Full-Random-DSG and Exc-RandomDSG, respectively). The results of these comparisons, as shown in Table 11, demonstrate the rationale of our approach. We recognize that the strategies we employed for the selection and grouping of discourse relations exhibit certain limitations. The specific grouping strategies may confuse the true features of categories, which merit further investigation in future research endeavors