PITA: Prompting Task Interaction for Argumentation Mining (2024)

Abstract: Argumentation mining (AM) aims to detect the arguments and their inherent relations from argumentative textual compositions. Prior methods are afflicted by a sequential feature decoding paradigm, wherein they initially address the features of argumentation components (ACs) for argumentative relation type classification (ACTC) subtask. Then, these features are amalgamated in pairs for argumentative relation identification (ARI) subtask. Finally, the AC pairs and ascertained pertinent relations are employed for argumentative relation type classification (ARTC) subtask. However, these methods merely rely on a shared encoder to implicitly capture the interactions of the three subtasks, which cannot explicitly and comprehensively model the inter-relationship among subtasks. In this paper, we propose a novel method PITA for PromptIng Task interAction to model the inter-relationships among the three subtasks within a generative framework. Specifically, we employ a dynamic prompt template to indicate all ACs and AC pairs in the three subtasks. Then, we construct an undirected heterogeneous graph to capture the various relationships within and between ACs and AC pairs. We apply the Relational Graph Convolutional Network (RGCN) on the graph and inject the task interaction information into the soft prompts with continuous representations. PITA jointly decodes all ACs and AC pairs using the prompt template with task interaction information, which thus explicitly and comprehensively harmonizes the information propagation across the three subtasks. Extensive experiments show PITA achieves state-of-the-art performances on two AM benchmarks.

heading: 1 Introduction
text: Argumentation mining (AM) (Lawrence and Reed, 2020) aims to detect argumentation structures in
* Equal Contribution. † Bin Liang, Min Yang and Ruifeng Xu are correspond-
ing authors.
an argumentation text by identifying the arguments and the relations between them. Generally, AM involves four subtasks, including (1) Argument component segmentation (ACS) which extracts argument components (ACs) from an argumentative text; (2) argument component type classification (ACTC) that classifies the type of each AC (i.e., Claim or Premise); (3) argumentative relation identification (ARI) that identifies the argumentative relation (i.e., Relevant and No-Relevant) of AC pairs; (4) argumentative relation type classification (ARTC) that determines the type of the ARs (i.e., Support and Attack). We follow previous works (Potash et al., 2017; Kuribayashi et al., 2019; Bao et al., 2021a; Morio et al., 2022) and assume that the first subtask ACS has been completed, that is, ACs have been segmented, and focus on the other three subtasks. Figure 1 shows an example of an argumentative text and its structure, where the text is decomposed into five ACs and contains four AC pairs with ARs.
5036
Previous works (Kuribayashi et al., 2019; Bao et al., 2021a; Morio et al., 2022) usually employed multi-task learning to capture the relationships among the three subtasks, which have achieved remarkable progress for AM. Most of them utilized sequential decoding. Specifically, previous methods first separately tackled the ACs for ACTC. Then, they exploited the AC representations to model the relation between AC pairs for ARI. Finally, the AC pairs with identified relevant relations were used to classify the relation types for ARTC.
However, these methods, merely relying on a shared encoder, only implicitly capture task interactions and limit explicit, comprehensive intertask modeling. Concretely, in sequential decoding, the information only explicitly flows from the ACTC decoder to ARI and ARTC decoders, not vice versa, potentially misguiding subsequent predictions if the former task decoding is inaccurate. For instance, the state-of-the-art model ST (Morio et al., 2022) misclassified the AC2 and AC4 as two Claims with a high degree of confidence. The pair decoder for ARI identifies the relation of the AC pair (AC2, AC4) as Non-Relevant, which is an incorrect answer, misleading the relation type for the ARTC subtask. The imbalanced information flow (i.e., from ACs in ACTC to AC pairs in ARI and ARTC as shown in Figure 2(a)) might have induced this error. Therefore, we argue that it may be important to model the inter-relationships among ACTC, ARI and ARTC for AM.
To address the aforementioned issue, we propose a novel method PITA for PromptIng Task interActions to model the inter-relationships among ACTC, ARI and ARTC. Our framework employs a generative encoder-decoder pre-trained language model (PLM), where the encoder focuses on the representation learning of ACs and AC pairs in the input text. The decoder equipped with a joint feature decoding mechanism facilitates the learning of task interaction patterns.
We recognize that subtasks are interconnected by the interactions among these task-specific representations of ACs or AC pairs. Specifically, to explicitly model the inter-relationship among subtasks and prompt the PLM, we first devise a dynamic prompt template to indicate all ACs and AC pairs. Considering these multiplex interrelationships among ACs and AC pairs for the three subtasks, we construct an undirected heterogeneous graph. In the graph, the ACs and AC pairs serve as nodes, while their interrelations form the edges.
Then, we apply Relational Graph Convolutional Networks (RGCN) (Schlichtkrull et al., 2018) over the graph to capture task interaction information among ACs and AC pairs. Finally, we inject the information into the soft prompt with continuous representation as the input of the decoder for jointly decoding all ACs and AC pairs for ACTC, ARI and ARTC simultaneously, rather than in separate steps.
The joint decoding mechanism explicitly incorporates task interactions, addressing the issues arising from sequential decoding. This method achieves equilibrium in the dissemination of information between individual ACs and their corresponding AC pairs. As shown in Figure 2(b), it not only utilizes inherent information existing amidst these ACs in ACTC for AC pairs in ARI and ARTC like previous methods (i.e., information flow from AC to AC pairs). But also it makes an AC assign heightened attention to evaluating its suitability for integration into an AC pair, transcending the scope of its self-contained informational content (i.e., information flow from AC pairs to ACs). In addition, different from the vanilla autoregressive generation paradigm (Lewis et al., 2019), our prompt template with the joint decoding mechanism only needs to be fed into the decoder one time and processed in parallel on GPUs.
In summary, our main contributions are as follows. (1) We introduce a novel method termed Task Interaction-Based Prompt Tuning (PITA) for AM. It employs prompt tuning in a generative framework catering to the multifaceted demands of multitask AM. (2) We introduce a graph-based method to learn task interaction and information injection, devised to effectively represent the multifaceted relationships that exist between ACs and their respective pairs. (3) We conduct extensive experiments on two AM benchmarks. Experimental results exhibit significantly better performance for AM.
10

heading: 2 Task Definition
text: Given the input argumentative text W = {w1, w2, . . . , wn}, the goal of AM is to identify a series of ACs X = (x1, x2, . . . , xm) from it. Following previous work (Kuribayashi et al., 2019; Morio et al., 2022), we assume that the text span (b1i , b 2 i ) of each AC xi is given, which contains the beginning word index b1i and the ending word index b2i . Specifically, for each AC xi, an AM system first needs to predict its AC type yACTCi ∈ YACTC (e.g., Claim and Premise). Then, it identifies the AR yARI(i,j) ∈ YARI (e.g., Rel and No-Rel) between two ACs xi and xj . Finally, the type of each AR yARTC(i,j) ∈ YARTC (e.g., Support and Attack) needs to be classified.

heading: 3 Methodology
text: Our proposed method, PITA, employs BART (Lewis et al., 2019) as the base model, and augments it with task interaction. BART is a standard Transformer-based PLM consisting of an encoder and a decoder: BART = [BARTenc,BARTdec]. The encoder is employed for representation learning of ACs and AC pairs, and the decoder is used for task interaction pattern learning among ACTC, ARI and ARTC. To enable BART to learn task interaction patterns, we first design a task interaction prompt template for the input text, as described in Section 3.2.1. The prompt template contains placeholders, which are essentially learnable vectors, serving as the reserved interface for task interaction information. Then, a task interaction graph is constructed and modeled through an RGCN (Section 3.2.2). The resulting node representations are integrated into BARTdec by
the placeholders of the prompt template. In this way, the decoding process can be guided by the task interaction pattern (Section 3.2.3). Figure 3 illustrates the architecture of PITA.

heading: 3.1 Text Encoding
text: Given a piece of argumentative text W with AC spans, this module generates the AC and AC pair representations. Specifically, we feed W into BART-Encoder to obtain the context representation matrix HW ∈ Rn×d, where d denotes the dimension of hidden states of BART.
HW = BARTenc(W ) (1)
We use mean-pooling operation over HW to obtain the input text representation hW . The representation of each AC xi is derived by mean-pooling over HW : hi = 1b2i−b1i+1 ∑b2i k=b1i
HWk . Then, the representation of each AC pair is calculated by averaging the representations of the two ACs in the pair: h(i,j) = (hi + hj)/2. Subsequently, the output of the BART-encoder will be used as the input of the BART-decoder.

heading: 3.2 Task Interaction Prompt
text: To facilitate task interaction learning, we inject continuous prompt tokens into the decoder of BART.

heading: 3.2.1 Prompt Template
text: We devise a task interaction prompt template to produce the continuous prompt tokens for BART. In the prompt template, we employ different placeholders 1 for different ACs in the input text. Although both the ARI and ARTC subtasks use AC
1The placeholder, also known as virtual word, is implemented using specific tokens like the eos token < s > in the BART vocabulary, and possess learnable vectors.
pairs to complete objectives, the task goals are different (i.e., relation identification for ARI and relation type classification for ARTC). We use two separate placeholders for the same AC pair for ARI and ARTC. In particular, given an argumentative text W with m ACs for ACTC, m × (m − 1)/2 AC pairs for ARI and m× (m− 1)/2 AC pairs for ARTC, the prompt template can be defined as:
T = Concat([D], TACTC , TARI , TARTC) TACTC = [AC1] . . . [ACm] TARI = [P(1,2)] . . . [P(m−1,m)] TARTC = [Q(1,2)] . . . [Q(m−1,m)]
where [D] denotes the input text placeholder, providing global information (e.g., topics) for subsequent ACs and AC pair placeholders. This also proves effective in other tasks such as dialogue generation (Li et al., 2021). [ACi], [P(i,j)] and [Q(i,j)] denote the placeholders of AC xi, AC pair (xi, xj) and AC pair (xi, xj) for ACTC, ARI and ARTC subtasks, respectively. Instead of using a fixed number of template tokens for all instances, our dynamic prompt template has different template tokens for different instances. Each token is infused with knowledge specific to these subtasks and designed to facilitate interaction among subtasks.

heading: 3.2.2 Task Interaction Graph Construction
text: Since the BART-decoder is an autoregression module, it only constructs the directed interaction of left-to-right among the prompt tokens and lacks their mutual interactions. To effectively capture the task interactions and balance the information propagation between ACs and AC pairs for the three subtasks, we construct an undirected heterogeneous graph.
The heterogeneous graph has four kinds of nodes corresponding to the placeholders in the prompt template: the input text placeholder [D] as the global node, the AC placeholders [AC∗] as the AC nodes and the two AC pair placeholders [P∗] and [Q∗] as the two kinds of AC pair nodes. Moreover, there are three kinds of inter-node edges in our graph:
• Global Edge: The global node is connected to all other nodes. It can transmit the global information in the input text to other nodes.
• Intra-Task Edge: In each subtask, all nodes are fully connected, i.e., AC nodes connect AC nodes in ACTC and AC pair nodes connect AC pair nodes in ARI (ARTC). These edges
can help the AC/pair nodes access contextual information.
• Inter-Task Edge: Two nodes from two different subtasks are connected if they have an inclusion relationship. For example, the AC pair node [P(i,j)] ([Q(i,j)]) in ARI (ARTC) links individual AC nodes [ACi] and [ACj ] as the AC pair (xi, xj) includes AC xi and xj . The AC pair node [P(i,j)] in ARI is connected with the AC pair node [Q(i,j)] in ARTC. They are the primary way for ACs and AC pairs to interact with each other. It not only helps the AC pair nodes to invoke the inherent information of ACs. But also the AC nodes can interact with AC pair nodes to transmit argumentation relation information of AC pairs to themselves.

heading: 3.2.3 Task Interaction Information Injection
text: To make full use of the graph-based task interaction pattern, we inject the task interaction information into template embedding 2. Specifically, we first feed the whole prompt template into the embedding layer of BART, which embeds input sequence T into the embedding space ET = [eD, e1, . . . , em, e P (1,2), . . . , e P (m−1,m), e Q (1,2), . . . , eQ(m−1,m)] , where e D denotes the embedding of input text placeholder [D]. ei, eP(i,j) and e Q (i,j) are the embeddings of AC and AC pair placeholders [ACi], [P(i,j)] and [Q(i,j)]s, respectively.
Then, we apply an RGCN (Schlichtkrull et al., 2018) on our undirected heterogeneous graph to model the task interaction pattern. Given a node u ∈ T at the l-th RGCN layer, the information interaction and aggregation operation is defined as follows:
el+1u = ReLU(e l u +
∑
r∈R
∑
v∈Nr(u)
1
|Nr(u)| W lre l v + b l r) (2)
where Nr(u) denotes the neighbors for node u connected with the edge of type r, ReLU is the ReLU activation function, W lr and b l r are the trainable parameters. For the first RGCN layer, we adopt the embeddings of placeholders in the template T to initialize node features. Finally, we select the representation of all nodes of the last layer L as the
2We also consider directly putting the template embedding into BART-Decoder and take the output hidden states of the decoder as the input of the RGCN to incorporate task interaction information. The detailed analysis can be seen in the ablation study.
updated prompt template representations, which aggregate task interaction information from a certain heterogeneous graph.
Although the prompt and context information would interact with each other at the cross-attention layers in the decoder, simply feeding the updated template placeholder features into the decoder makes it hard to focus accurately on the correct segment of input text for ACs and AC pairs. Therefore, we add the context representations of ACs and AC pairs to the updated template representations to obtain the context-specific prompt embeddings ÊT = [êD, ê1, . . . , êm, ê P (1,2), . . . , ê P (m−1,m), ê Q (1,2), . . . , êQ(m−1,m)] , where ê D = eD,L+hW , êi = eLi +hi, êk(i,j) = e k,L (i,j) + h(i,j) and k ∈ {P,Q}.
After that, we feed the prompt representations into the BART decoder to get the output hidden states h0i , h 1 (i,j) and h 2 (i,j) of the placeholders [ACi], [P(i,j)] and [Q(i,j)]. Recall the joint decoding process of ACs and AC pairs from the prompt. It is obvious that the interaction among different subtasks is considered under this paradigm. In this way, our method can solve the problems in sequential decoding and balance the information between ACs and AC pairs.

heading: 3.3 Objective Function
text: The training objective of PITA is to generate outputs that replace the placeholders in the prompt template with gold labels. For ACTC, PITA is expected to derive the argumentation type of the AC xi using the hidden state of placeholder [ACi]. PITA employs the placeholder [P(i,j)] to determine the relation between AC pair (xi, xj) for ARI. For ARTC, PITA uses the placeholder [Q(i,j)] to predict the relation type between AC pair (xi, xj). Formally, we create a learnable label word vector vi for each label yi in each subtask. We utilize crossentropy functions as the learning objectives of the ACTC, ARI and ARTC subtasks, which are defined as follows:
LACTC = − m∑
i=0
log(ŷACTCi )
LARI = − m∑
i=0
m∑
j=i+1
log(ŷARI(i,j))
LARTC = − m∑
i=0
m∑
j=i+1
log(ŷARTC(i,j) )
(3)
where ŷACTCi = v 0 kh 0 i , ŷ ARI (i,j) = v 1 kh 1 (i,j) and ŷARTC(i,j) = v 2 kh 2 (i,j) are the predicted probability
of ground truth labels yk of the AC xi for ACTC, AC pair (xi, xj) for ARI and AC pair (xi, xj) for ARTC subtasks, respectively. v0k, v 1 k and v 2 k are the k-th label vector in ACTC, ARI and ARTC. We train PITA by jointly optimizing the three subtasks. The total training object is defined as follows:
L = LACTC + LARI + LARTC (4)

heading: 3.4 Efficiency Considerations
text: PITA has three components including a BART, a RGCN and a prompt template. The RGCN is required to model the task interaction graph, which has a time complexity O(m4) and could cause efficiency considerations. In practice, this issue is minor for our experiments on the two datasets (PE and CDCP) and real scenarios, because the number 3 of ACs in argumentation texts is usually relatively small scales. Noted our prompt template with the joint decoding mechanism only needs to be fed into the decoder one time and processed in parallel. It is more efficient compared with the autoregressive paradigm (Lewis et al., 2019) which generates tokens sequentially during inference. We present the time cost of PITA in Appendix A.4.

heading: 4 Experimental Setup
text: Datasets To evaluate the effectiveness of our PITA model, we conduct extensive experiments on two widely-used AM datasets, PE (Stab and Gurevych, 2017) and CDCP (Park and Cardie, 2018) following the official split. The detailed statistics of PE and CDCP are summarized in Appendix A.1.
Evaluation Metrics We employ the same evaluation metrics with the previous works (Kuribayashi et al., 2019; Liu et al., 2022), including F1 score and macro averaged score (denoted as Macro). We adopt the macro averaged score for ACTC and ARTC and calculate F1 scores for determining the relevant (Rel) AR between ACs for ARI following (Morio et al., 2022).
Baselines We compare PITA with the following strong baseline models. Following previous works (Morio et al., 2022), for the PE dataset, we compare our model with six strong baselines, including Joint-ILP (Stab and Gurevych, 2017),
3The average number of ACs in the PE and CDCP samples is 3.5 and 6.5, respectively.
Joint-PN (Potash et al., 2017), LSTM+dist (Kuribayashi et al., 2019), BERT-Trans (Bao et al., 2021a) BART (Lewis et al., 2019) and ST (Morio et al., 2022). For the CDCP dataset, we compare our model with five strong baselines, which are DR-LG (Galassi et al., 2018), SSVM-strict (Niculae et al., 2017), TSP-PLBA (Morio et al., 2020), BERT-Trans (Bao et al., 2021a), BART (Lewis et al., 2019) and ST (Morio et al., 2022).
Implementation Details We use PyTorch to implement the proposed framework based on Transformers (Wolf et al., 2020) on NVIDIA TESLA A100-PCIE-40GB. Our model is optimized using AdaW (Loshchilov and Hutter, 2017) with the learning rates of 3e-5 and weight decay of 1e-2 on both PE and CDCP datasets. We set the batch size to 4 on both PE and CDCP datasets. For both datasets, we adopt dropout (Srivastava et al., 2014) with a dropout rate of 0.1 to avoid overfitting. We set the layer number L of RGCN to 1 because of its best performance. All experiments are performed 5 times with different random seeds, and the evaluation scores are averaged. Our code is available at https://github.com/syiswell/PITA.

heading: 5 Experimental Results

heading: 5.1 Performance Comparison
text: The comparison results on PE and CDCP datasets are summarized in Table 1 and Table 2, respectively. We can observe that PITA achieves the best performance on both datasets. On the PE dataset, our PITA model outperforms state-of-the-art (SOTA) model ST by 4.2% and 2.1% on ARI and ACTC subtasks in terms of F1 and Macro, respectively. We observe similar trends for the CDCP dataset. Our model brings 4.7% improvement of F1 score for ARI and 3.4% improvement of Macro score for ARTC. We argue that task interaction learning by joint decoding manner plays an important role in PITA, balancing the information obtained by ACs and AC pairs.
In addition, we also observe that the featurebased models (i.e., Joint-ILP and St-SVM-strict) perform poorly since they heavily rely on feature engineering. The BERT- and Longformer-based neural network (i.e., BERT-Trans and ST) consistently outperforms the LSTM-based baselines (i.e., LSTM+dist and TSP-PLBA). This may be because BERT and Longformer-based methods can exploit rich knowledge from PLMs trained on large-scale general corpora. Our PITA model performs better than the strong BERT- and Longformer-based models by modeling task interaction patterns to effectively capture the rich dependency information within and across the ACs and AC pairs.

heading: 5.2 Ablation Study
text: To analyze the impact of different components in PITA, we conduct ablation studies and report the results in Table 3. We can observe that by removing the whole task-interaction graph (w/o TIG), the model cannot access the connectivity of the ACs and AC pairs, with significant performance degradation. It is worth noting that removing the whole task-interaction graph (w/o TIG) is equal to only using the prompt template, which is ultimately close
to vanilla BART. In addition, to further illustrate the effectiveness of task interaction information injection, we put the task interaction graph behind the BART-Decoder and the hidden states of the decoder as the input of the RGCN (denoted by PB TIG). We can observe that PB TIG leads to a significant decrease in performance. This is because the PB TIG separately decodes the representations of ACs and AC pairs in the decoder, which cannot efficiently utilize the knowledge within the PLM to obtain high-quality AC and AC pair representations.
To examine the importance of different components in the task interaction graph, we consider removing the different nodes and edges. We can see that after removing the global node with the global edge (w/o G-Edge), the model performance drops, showing the effectiveness of the global information for ACs and AC pairs. Removing the intra-task edge (w/o IRT-Edge) degrades the performance, verifying that contextual information is important for AM. Removing inter-task edge (w/o IET-Edge) leads to performance drops, demonstrating that task interaction pattern learning facilitates a balanced propagation of information between ACs and their corresponding AC pairs.

heading: 5.3 Adaptability Experiment
text: Inspired by the recent success of Large Language Models (LLMs) (Min et al., 2023), we conduct additional experiments on two prominent LLMs, namely ChatGPT-3.5-Turbo and LLAMA2. We employ the natural language prompt approach like (Madaan et al., 2022; Li et al., 2023) and fewshot in-context learning (here 3-shot is used due to the limitation of input length) in ChatGPT-3.5-
Turbo and LLAMA2 for AM. Figure 6 displays the format of the natural language prompt for AM. To evaluate the adaptability of our method, we adapt our approach to fine-tuning LLAMA2 using LORA (Hu et al., 2021) (denoted by LLAMA2PIAT). Besides, we fine-tune LLAMA2 with the natural language prompt approach (denoted by LLAMA2-FT) and LLAMA2-PIAT without the task interaction graph using LORA as baselines. Each fine-tuning method uses a batch size of 1. The results are presented in Table 4.
We observe that few-shot-based methods (i.e., LLAMA2 and ChatGPT-3.5-Turbo) significantly underperform fine-tuning methods (i.e., PITA, LLAMA2-PITA and LLAMA2-FT) on PE and CDCP. Among these fine-tuning approaches, our PITA framework (LLAMA-PITA) outperforms the w/o TIG as well as the LLAMA-FT which is finetuned using natural language prompt, validating the effectiveness of our task interaction learning. In addition, LLAMA-PITA performs worse than PITA using BART as the base model. This is because LLAMA-PITA has a large number of parameters (including trainable and non-trainable parameters) while the AM datasets have a small number of samples, which leads to severe overfitting.

heading: 5.4 Impact of Different Templates
text: We explore how different types of prompts affect the performance in this section, as shown in Figure 4. We first compare two template variants with our method: PITA-S, which uses an identical placeholder for different ACs (AC pairs) in each subtask, and PITA-M, where each AC and AC pair followed an [MASK] token for prediction, simulating the pretrained objective of BART. Furthermore,
We argue that there are task order biases in our PITA model due to the autoregressive generative paradigm. Thus, we investigate the effect of task order in the template and explore five variants (denoted by PITA-ICR, PITA-IRC, PITA-CRI, PITARCI and PITA-RIC) that rearrange the placeholder positions for the ACTC (C), ARI (I), and ARTC (R) tasks. The formats of all prompt templates are detailed in Table 7.
We find that (1) our template outperforms other template variants; (2) PITA-S performs worse than ours. With the same placeholders in each subtask, where the template neither contains specific knowledge about ACs or AC pairs nor learns task interaction patterns during training; (3) PITA-M achieves worse results than ours, demonstrating our ACs and AC pairs placeholders can be better used for task-specific prediction by injecting task interaction information, whereas the extra [MASK] simply contains the priori knowledge learned during pretraining; (4) Unsurprisingly, PITA-ICR, PITA-IRC, PITA-CRI, PITA-RCI and PITA-RIC obtain worse performance than ours, which verifies that the task order of ACTC, ARI and ARTC is better setting. This setting is widely used in previous works.

heading: 5.5 Impact of Different Connections of Task Interaction Graph
text: In the task interaction graph, we connect nodes according to the strategy described in Section 3.2.2, but this connection is not unique. To assess the effectiveness of the proposed connection strategy, we experiment with two different variants. The first variant, termed Random connection (RC), randomly links two nodes. The second variant, named Full connection (FC), links each node pair. The results are presented in Table 5. RC outperforms FC yet falls short of our strategy, indicating that indiscriminate node connections can introduce noisy information. Our connection strategy considers not only the contextual information within each subtask and the interactions across the three subtasks. This ensures a balanced information flow between Argument Components (ACs) and AC pairs, leading to superior results.

heading: 5.6 Case Study
text: We analyze two examples selected from the benchmark corpus to demonstrate the effectiveness of task interaction for balancing information among three subtasks, which is shown in Figure 5. Our task interaction graph makes ACs and AC pairs
interact with each other. It not only helps the AC pair nodes to invoke the type information of ACs and itself. But also the AC nodes can interact with AC pair nodes to transmit argumentation relation information of AC pairs to themselves. In the first example, although ST classifies all the correct ACs, it cannot couple the correct pair (AC3, AC4). By considering the argumentation relationship among ACs, PITA avoids this situation. For the second example, ST and PITA both classify the wrong type for AC4 in ACTC. In ARTC, the sequential encoding method ST cannot couple the pair (AC2, AC4), resulting in one undetected pair. On the contrary, PITA can avoid them and correct the pair type by task interaction pattern learning and the balance of information flow between AC and AC pairs.

heading: 6 Related Works

heading: 6.1 Argumentation Mining
text: Argumentation mining (AM) aims to identify and extract the argumentation structures from argumentative texts automatically (Lawrence and Reed, 2020; Bao et al., 2021b; Cheng et al., 2021; Sun et al., 2022; Guo et al., 2023; Chen et al., 2023). Early works (Peldszus and Stede, 2015; Persing and Ng, 2016; Stab and Gurevych, 2017; Afantenos et al., 2018) applied methods like minimum spanning trees (MST) and integer linear programming (ILP) with discrete features, focusing heavily on feature engineering, which is both laborintensive and time-consuming. With the rise of deep learning, Potash et al. (2017) introduced a sequence-to-sequence model with pointer networks and attention for AM. Kuribayashi et al. (2019) added linguistic clues to enhance AM performance, while Niculae et al. (2017) used SVMbased structured learning for both tree and non-tree data, albeit requiring argumentation-specific factor graph designs. Galassi et al. (2018) and Morio et al. (2020) introduced residual networks and taskspecific modules for AM, tackling tree and non-tree structures separately. Bao et al. (2021a) proposed a neural transition-based model using BERT for
ACTC and transforming ARI into action prediction, and Morio et al. (2022) used a Longformer-based model with biaffine function for AM. Different from previous works, we explicitly model the task interactions by constructing the complex relations within and between ACs and AC pairs for AM through a generative framework.

heading: 6.2 Prompt Tuning
text: Prompt tuning (Liu et al., 2023) has attracted much attention in the field of natural language processing (NLP), such as text classification (Schick and Schütze, 2021; Wang et al., 2022), text generation (Li and Liang, 2021) and information extraction (Ma et al., 2022). Existing studies on prompt tuning learning mainly focus on discrete and continuous prompts. The former designs text-based prompts (Jiang et al., 2020; Gao et al., 2020; Schick and Schütze, 2020), while the latter prepend a learnable prompt vector to word embeddings (Lester et al., 2021). We adopt the latter because of its flexibility and extensiveness. To the best of our knowledge, our PITA is the first work to use prompt tuning within a generative framework for AM.

heading: 7 Conclusion
text: In this paper, we proposed a novel model PITA, which prompted task interaction to model the relationships within and between ACs and AC pairs through a joint decoding mechanism encompassed within a generative framework. To explicitly model the interaction among three subtasks, we devised a dynamic prompt template to prompt all ACs and AC pairs in the three subtasks. Then, we constructed an undirected heterogeneous graph to capture the comprehensive relationships within and between ACs and AC pairs. Experimental results
on two benchmarks showed that our method outperformed strong baselines significantly.
Limitation
To point out future research direction for AM, we perform an error analysis on 100 cases where our PITA made mistakes for ACTC, ARI and ARC subtasks. In the ACTC subtask, we discover that there is a type bias for different positions of ACs. For instance, PITA tends to predict the first AC as “Claim” and the last AC as “Promise”. This is because PITA overfits the type distribution of ACs’ different positions based on the prompt placeholders. Maybe we can adopt a debias approach to alleviate this issue. For ARI, we find that PITA captures the connection between long-distance ACs so excessively that it identifies additional AC pairs. In the argumentation structure, AC pairs with relationships should not be too far apart, or it won’t follow the argumentation structure habits of human texts. Therefore, we suggest that during ARI tasks, incorporating distance loss between ACs can guide the model to focus more on the connection between ACs with moderate distances, which could potentially address the issue. In the ARTC task, the most serious problem is the error identification of AC pairs, resulting in incorrect AC pair type classification. In addition, there is a serious class imbalance problem that induces incorrect model predictions.
Last, we argue that there are order biases in our PITA model as described in Table 4 due to the autoregressive generative paradigm. In particular, the order of the placeholders in the prompt template is fixed, but there are actually no order relations between these placeholders. Although we mitigate the order bias between tasks using task interaction learning and experimentally validate the effective-
ness of our task interaction learning. However, two directions worth exploring are combining multiple templates with different task orders to eliminate order bias between tasks and the fact that order bias within each task needs to be taken into account, even if it may be increases the complexity of the model. Therefore, the tradeoff between order modeling and complexity is what we need to explore. Additionally, a new method that is free from the influence of order bias is more desirable.

heading: Acknowledgements
text: We thank the anonymous reviewers for their valuable suggestions to improve the quality of this work. This work was partially supported by the National Key Research and Development Program of China (2022YFF0902100), the National Natural Science Foundation of China (62176076), Natural Science Foundation of Guangdong (2023A1515012922), the Shenzhen Foundational Research Funding (JCYJ20210324115614039, JCYJ20220818102415032), the Major Key Project of PCL2021A06, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005).

heading: A Appendices
text: A.1 Data statistics We conduct experiments on two popular benchmark datasets.
• PE (Stab and Gurevych, 2017): The PE dataset contains 420 essays with 1833 paragraphs. There are three types of ACs (i.e., MajorClaim (MC), Claim and Premise) and two types of ARs (i.e., Support and Attack). Each AC has at most one outgoing AR so the argumentation graph of the paragraph can be either directed trees or forests. In addition, we extend each AC by including its argumentative marker following previous works (Kuribayashi et al., 2019; Bao et al., 2021a). We split this dataset into a training set of 1464 ACs and a testing set of 369 ACs, and randomly choose 10% of the training set as the validation set, which is consistent with previous works (Kuribayashi et al., 2019; Bao et al., 2021a).
• CDCP (Park and Cardie, 2018): The CDCP dataset contains 731 paragraphs. All ACs are classified into five types: Value, Policy, Testimony, Fact and Reference. The ARs have two types: Reason and Evidence. Each AC may have several outgoing ARs, thus the argumentation graph is of non-tree structure. The whole dataset is partitioned into a training set of 581 ACs and a testing set of 150 ACs. We randomly choose 10% of the training set as the validation set following (Bao et al., 2021a; Morio et al., 2022).
The statistics of the two datasets are summarized in Table 6.
A.2 Baselines Following previous works (Morio et al., 2022), for the PE dataset with tree structure, we compare our model with six strong baselines:
• Joint-ILP (Stab and Gurevych, 2017): This model optimizes argument component types (ACTC) and argumentative relations (ARI) using Integer Linear Programming.
• Joint-PN (Potash et al., 2017): It applies a Pointer Network with attention mechanism to jointly learn ACTC and ARI.
• LSTM+dist (Kuribayashi et al., 2019): This work first introduce LSTM-minus-based span representation with pretrained ELMO embedding for of AM.
• BERT-Trans (Bao et al., 2021a): This model employs neural transition-based model by generating a sequence of actions for argumentation mining.
• BART (Lewis et al., 2019): The model only uses a BART and converts the AM into a sequence generative formulation. The format of the target sequence is similar to our prompt format.
• ST (Morio et al., 2022): This method employs Longformer (Beltagy et al., 2020) and biaffine function for AM.
For the CDCP dataset with non-tree structure, we compare our model with six strong baselines, which are:
• DR-LG (Galassi et al., 2018): This method explores the use of residual networks with link-guided training to jointly learn ACTC, ARI and ARTC.
• SSVM-strict (Niculae et al., 2017): This method is a variant of structured SVM with strict factor graph for both ACTC and ARI.
• TSP-PLBA (Morio et al., 2020): The model incorporates task-specific parameterization to encode ACs and proposition-level biaffine attention to capture the structure of argumentation corpus.
• BERT-Trans (Bao et al., 2021a): This model employs neural transition-based model by generating a sequence of actions for argumentation mining, which is also the current state-ofthe-art method on the CDCP dataset.
• BART (Lewis et al., 2019): The model only uses a BART and converts the AM into a sequence generative formulation. The format of the target sequence is similar to our prompt format.
• ST (Morio et al., 2022): This method employs Longformer (Beltagy et al., 2020) and biaffine function for AM.
A.3 Impact of Different Templates
We explore how different types of prompts affect the performance in this section, as shown in Figure 4. We first compare two template variants with our method: a template (denoted by PITA-S) with the same placeholder for different ACs (AC pairs) in each subtask, a template (denoted by PITA-M) with each AC and AC pair following an added [MASK] for prediction to simulate the pretrained objective of BART. In addition, We argue that there are task order biases in our PITA model due to the autoregressive generative paradigm. Thus, we investigate the effect of task order in the template and explore five templates (denoted by PITA-ICR, PITA-IRC, PITA-CRI, PITA-RCI and PITA-RIC) in which the positions of placeholders in the three tasks are swapped. The format of all prompt templates can be viewed in Table 7.
A.4 Computational Cost
We investigate the computational cost of baseline methods and our PITA model in training and inference. For a fair comparison, all these models use the same batch size of 4 in training and inference. Table 8 shows the training time and inference time on the PE and CDCP. PITA has a competitive efficiency compared to the SOTA model (i.e., ST) for training, while faster than ST for inference. For example, PITA shows a decrease of 1.54s and 0.49s in inference time for all samples in PE and CDCP, respectively, compared to ST, which verifies the efficiency of our method. In addition, during inference, our PITA is more efficient than the generation-based baseline (i.e., BART) by a fac-
tor of 6-18 owing to our task interaction prompt learning, although minimal overhead is added to the training process. Comparing PITA with w/o TIG, the task interaction graph introduces a small amount of extra time (almost 1.8s on PE and 1.2s on CDCP for one epoch) in both the training and inference phases (0.28 seconds on PE and 0.21 seconds on CDCP for all instances), which is acceptable in practice.

heading: Future Work
text: Limitation
To point out future research direction for AM, we perform an error analysis on 100 cases where our PITA made mistakes for ACTC, ARI and ARC subtasks. In the ACTC subtask, we discover that there is a type bias for different positions of ACs. For instance, PITA tends to predict the first AC as “Claim” and the last AC as “Promise”. This is because PITA overfits the type distribution of ACs’ different positions based on the prompt placeholders. Maybe we can adopt a debias approach to alleviate this issue. For ARI, we find that PITA captures the connection between long-distance ACs so excessively that it identifies additional AC pairs. In the argumentation structure, AC pairs with relationships should not be too far apart, or it won’t follow the argumentation structure habits of human texts. Therefore, we suggest that during ARI tasks, incorporating distance loss between ACs can guide the model to focus more on the connection between ACs with moderate distances, which could potentially address the issue. In the ARTC task, the most serious problem is the error identification of AC pairs, resulting in incorrect AC pair type classification. In addition, there is a serious class imbalance problem that induces incorrect model predictions. Last, we argue that there are order biases in our PITA model as described in Table 4 due to the autoregressive generative paradigm. In particular, the order of the placeholders in the prompt template is fixed, but there are actually no order relations between these placeholders. Although we mitigate the order bias between tasks using task interaction learning and experimentally validate the effective-
ness of our task interaction learning. However, two directions worth exploring are combining multiple templates with different task orders to eliminate order bias between tasks and the fact that order bias within each task needs to be taken into account, even if it may be increases the complexity of the model. Therefore, the tradeoff between order modeling and complexity is what we need to explore. Additionally, a new method that is free from the influence of order bias is more desirable