DiscoPrompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition
Year: 2023

Abstract:
Implicit Discourse Relation Recognition (IDRR) is a sophisticated and challenging task to recognize the discourse relations between the arguments with the absence of discourse connectives. The sense labels for each discourse relation follow a hierarchical classification scheme in the annotation process (Prasad et al., 2008), forming a hierarchy structure. Most existing works do not well incorporate the hierarchy structure but focus on the syntax features and the prior knowledge of connectives in the manner of pure text classification. We argue that it is more effective to predict the paths inside the hierarchical tree (e.g., “Comparison -> Contrast -> however”) rather than flat labels (e.g., Contrast) or connectives (e.g., however). We propose a prompt-based path prediction method to utilize the interactive information and intrinsic senses among the hierarchy in IDRR. This is the first work that injects such structure information into pre-trained language models via prompt tuning, and the performance of our solution shows significant and consistent improvement against competitive baselines.

heading: 1 Introduction
text: Discourse parsing is the task of automatically parsing discourse structure in a text, including the identification of discourse structure and the annotation of discourse relations (Li et al., 2022). Discourse Relation Recognition (DRR) is a crucial task in discourse parsing, recognizing relations between two arguments (i.e., sentences or clauses). It is vital for textual coherence and is considered as the essential step for many downstream tasks involving more context, such as question answering (Rutherford and Xue, 2015), text generation (Bosselut et al., 2018), and argument mining (Liu et al., 2021b). Explicit discourse relation recognition (EDRR) has already been demonstrated that utilizing explicit
∗ Equal contribution.
connectives information can effectively determine the discourse relation types (Varia et al., 2019). On the other hand, implicit discourse relation recognition (IDRR) is still challenging with the absence of connectives (Varia et al., 2019).
Traditional works on IDRR focus on syntax features, including word pairs (Lin et al., 2009; Varia et al., 2019) and other surface features (Ji and Eisenstein, 2015; Bai and Zhao, 2018). With deep neural networks and large language models (LLMs), different approaches pay much attention to text representations via attention (Liu and Li, 2016), pre-training (Shi and Demberg, 2019b), multi-task learning (He et al., 2020; Long and Webber, 2022), and prior knowledge (Liu et al., 2020; Zhou et al., 2022). But one important piece of information, i.e., the inherent discourse label hierarchy, is not fully investigated.
The sense labels for each discourse relation follow a hierarchical classification scheme in the annotation process of PDTB 2.0 framework (Prasad et al., 2008), forming a hierarchy structure. Figure 1 shows an example from PDTB 2.0 dataset (Prasad et al., 2008). It consists of two arguments (i.e., Arg1 and Arg2) and is annotated with relation senses, where the semantics of the top-level Comparison is further refined by the second-level Contrast. Besides, we list representative connectives (e.g., however) to help better understand the definitions and semantics of labels. LDSGM (Wu et al., 2022) uses
35
graph convolutional networks to encode the label dependencies into text representations, illustrating the importance of label structures on text representation learning and label prediction. However, such usage is not compatible with pre-training because it may significantly affect the representations from language models. Prompt tuning has shown its power in text classification without altering the representations from pre-trained language models, especially for low-resource scenarios (Schick and Schütze, 2021; Gao et al., 2021).
In this paper, we propose a prompt-based path prediction method, Discourse relation path prediction Prompt tuning model (DiscoPrompt 1), to utilize the hierarchy and intrinsic senses of labels in IDRR. Specifically, we transform the hierarchy in Figure 1 to “Comparison -> Concession -> if; · · · ; Temporal -> Synchrony -> when” as the hierarchical prompt and add it as the prefix of arguments to be classified. The dependencies of top and second-level relation senses are explicitly provided as the context. On the other hand, connectives are provided as the natural language explanations of labels to help the language models better adapt to the prior knowledge. We ask the LLMs to predict the label’s hierarchical path instead of the leaf label for IDDR, and we show such a way of providing the label hierarchy ahead of arguments significantly improves the IDRR performance. Our contributions are summarized as follows:
• This is the first work that injects labels’ hierarchical structure information and connectives into pre-trained language models via prompt tuning.
• We model the IDRR problem as the path prediction problem that predicts the joint probability of top-level relations, second-level types, and connectives at the same time.
• We conduct extensive experiments and thorough ablation studies to discuss the necessity and effectiveness of the label hierarchy and connectives. The results support our claims and the success of our proposed DiscoPrompt model.

heading: 2 Related Work
text: Prompt Tuning With LLMs, such as T5 (Raffel et al., 2020) and GPT-3 (Brown et al., 2020),
1The source code is available at https://github. com/HKUST-KnowComp/DiscoPrompt
prompt-based methods have attracted much attention in the field of natural language understanding (Schick and Schütze, 2021; Lester et al., 2021; Liu et al., 2022). Compared with fine-tuning, prompt tuning may have a better generalization on various tasks due to the aligned nature of language descriptions and answer semantics, e.g., classification problems (Gao et al., 2021; Wang et al., 2022a). At the same time, there are some efforts to leverage prompts with structural inputs for knowledge customization (Zhong et al., 2022). Injecting hierarchy information into prompts is also promising. For example, using top-level predictions to refine prompts of bottom levels can surpass soft prompts and hard prompts (Wang et al., 2022b). Nevertheless, how to employ LLMs to better involve hierarchy knowledge is still under investigation.
Implicit Discourse Relation Recognition It has been discovered that connectives can provide necessary clues in predicting discourse relations to achieve around 95% accuracy (Dai and Huang, 2019; Varia et al., 2019). However, the absence of connectives makes the prediction more challenging. Many efforts have been paid to explore the syntax through linguistic features (Rutherford and Xue, 2015; Ji and Eisenstein, 2015; Wang and Lan, 2016; Dai and Huang, 2018; Varia et al., 2019), attention (Liu and Li, 2016; Bai and Zhao, 2018), pre-training (Shi and Demberg, 2019b), knowledge transfer (Lan et al., 2017; Dai and Huang, 2019; He et al., 2020), etc. With the power of language models, connective prediction also illustrates its effectiveness in implicit relation prediction (Nguyen et al., 2019; Shi and Demberg, 2019a; Kishimoto et al., 2020; Kurfali and Östling, 2021). In addition, PCP (Zhou et al., 2022) shows the feasibility of combining label prediction and connective prediction under the manner of prompts. The latest methods reveal the significance of the label hierarchy of discourse relations. LDSGM (Wu et al., 2022) utilizes the graph convolutional networks to incorporate label dependencies into text representations, while ContrastiveIDRR (Long and Webber, 2022) leverages the sense hierarchy to obtain contrastive learning representation. However, these methods are incompatible with pre-training as they modify the representations from pre-trained language models. Therefore, this work investigates injecting the label dependencies information and connectives into pre-trained language models via prompt tuning with aligning the representations.

heading: 3 Method

heading: 3.1 Problem Definition
text: The sense labels in various levels of the Implicit Discourse Relation Recognition (IDRR) task naturally constitute a hierarchy, denoted as H. H is a hierarchical tree structure whose depth is d, with the root node in depth 0 and class sense of different levels distributed to the corresponding layer (i.e., from depth 1 to d) in this tree. Let class label set C to be ⋃dk=1 Ck where Ck = {ck1, · · · , cknk} is the label set of depth k, and nk is the number of classes at depth k. For example, the hierarchy H of PDTB 2.0 forms a tree with depth size 2, and the C2 corresponds to the label set of the second level, containing 11 class subtypes like Concession, Synchrony, etc. We can enrich the label hierarchy by adding a connective layer like in Figure 1. We adopt the Naive Bayes to compute the prior distribution Pr(c2|z) from the explicit relation data, where c2 ∈ C2 is a subtype, and z is the connectives. Figure 2 shows the heat map of highly frequent connectives. We can find that the connectives are the vital clue for discourse relations. Therefore, we select the most discriminative ones as C3. We do not observe significant improvement when adding more than one connective for each c2. Therefore, we summarize C for PDTB 2.0 in Table 1. Prior distributions and label words of CoNLL16 are shown in Appendix A.2.
In this task, given a data set D = {(xi, yi)} consisting of data instance xi = (a1i , a 2 i ) and label yi, where the a1i , a 2 i represent the argument 1 and argument 2 of respective instance i and the label yi is class label set. In our method, the class label set including d labels for d layers forms a path P in hierarchical tree H, instead of a single class label for a specific level. After predicting a path, the classes of various levels are the nodes lying in the predicted path. There-

heading: Top-level Second-level Connectives
text: fore, this task is to find out the optimal path:
P ∗i = argmaxPj Pr
( Pj | xi ) , (1)
where P ∗i is the optimal path and j indicates the j-th path among all paths.

heading: 3.2 T5 Backbone Model
text: T5 (Raffel et al., 2020) is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks. The unsupervised denoising training task required the model only to predict the masked consecutive spans of tokens. For example, the input “Thank you for inviting me to your party last week.” will be corrupted as “Thank you <X> me to your party <Y> week.” and the target is “ <X> for inviting <Y> last 
” 
 is the eos_token. In the supervised pre-trained task, the model was asked to perform the sequence-tosequence input-output mapping by specifying the task prefix (such as “translate German to English:" or “summarize:”). However, the specific textual prefix token is difficult to discover and requires a substantial amount of human effort. Hence, prefix tuning (Li and Liang, 2021) and prompt tuning (Lester et al., 2021) methods proposed to overcome this problem by relaxing the constraint of discrete textual tokens to continuous tunable ones.

heading: 3.3 Path Prediction Prompt Tuning Method
text: To predict the path P ∗i for each instance xi, we leverage a human-tailored template T (·) to convert the data instances to the prompt input x̃i = T (xi) and a verbalizer V(·) to map a set of words to class labels. Figure 3 illustrates the architecture of DiscoPrompt.
Structure-Aware Prompt The crafted template includes necessary discrete tokens, masked tokens, soft continuous tokens, and context with the hierarchy information. The first part of our prompt
template is the discrete tokens “The path is ” for eliciting the predicted path Pi. Then three [MASK] tokens are included: an [MASK] is inserted between two arguments for predicting the probability of decided connectives, and two [MASK]s form an edge “[MASK] -> [MASK]” for receiving the top and second-level class probabilities. We also added 20 learnable continuous tokens at the beginning of the template to effectively searching an optimal template. To better utilize the hierarchy information and senses of labels, we explicitly translate them into a tailored hierarchical tree prompt and insert it into the input. This hierarchical tree prompt is the discrete tokens appended ahead of the arguments as the context in natural language. Figure 8 in Appendix B.2 shows the details of the template.
Path Verbalizer A traditional verbalizer usually maps a label y to a single answer token z or a series of spans z1, z2, · · · greedily (Schick and Schütze, 2021; Liu et al., 2021a). We extend it by mapping a path P to three tokens, i.e. {Pj} → Z ×Z ×Z , where Z is the vocabulary. We denote the three [MASK] tokens as z1, z2, and z3. Then using the prompt template with three [MASK]s and the verbalizer V(·), the probability distribution over {Pj} can be formalized as the joint probabilities of z1, z2, and z3, i.e. Pr(Pj | x̃i) = Pr(V(Pj) | x̃i) = Pr(z1i = p j 3, z 2 i = p j 1, z 3 i = p j 2 | x̃i), where a path Pj consists of pj1 (the top-level), pj2 (the second-level), and p j 3 (the connective). Since T5 can synchronously predict masked tokens, the joint probability can be written as
Pr(Pj | x̃i) = 3∏
k=1
Pr(zki = v k(Pj) | x̃i), (2)
where vk(·) : {Pj} → Z is the submap of V(·) for the k-th [MASK]. The final learning objective of DiscoPrompt is to maximize
J = 1|D| ∑
(xi,yi)∈D log
3∑
k=1
Pr(zki = v k(Pj) | x̃i). (3)
Once we get the prediction of P∗i by choosing the maximum joint probability (i.e., path score) as Eq. (2), we can get the prediction of each level as Eq. (4).
ck ∗
i = argmax ck
Pr(ck | Pj , xi) · Pr(Pj | xi), (4)
where Pr(ck | Pj , xi) can be calculated by the prior probability (or simply set as 1.0).

heading: 4 Experimental Setting

heading: 4.1 Dataset
text: The experiments are conducted on two datasets, the PDTB 2.0 (Prasad et al., 2008) and the CoNLL2016 shared task (CoNLL16) (Xue et al., 2016), to validate the performance of our method. Both contain the Wall Street Journal (WSJ) articles, and the difference is the annotation and relation senses. We evaluate performance on PDTB 2.0 according to two different settings denoted as Ji (Ji and Eisenstein, 2015) and Lin (Lin et al., 2009) with 11 subtypes. The CoNLL-2016 shared task provides more

heading: 4.2 Implementation Details
text: We employ the T5 model (Raffel et al., 2020) as the backbone to implement DiscoPrompt and use the T5-large as the primary model for a fair comparison with extensive baselines. Generally, the overall configuration follows the setting in Lester et al. (2021), and we put more details of the configuration in Appendix A.2. We report the MacroF1 score and accuracy in experiments and ablation studies. A prediction is considered as correct whenever it matches one of the ground-truth labels. All experiments are conducted with 2 × NVIDIA V100 (32GB) except for the T5-11b scale on 2 × NVIDIA A6000 (48GB).

heading: 4.3 Baselines
text: This paper mainly adopts two categories of competitive baselines for the PDTB 2.0 dataset and the CoNLL-2016 shared task2. The first category is the previous state-of-the-art (SOTA) baselines, such as TransS (He et al., 2020), BMGFRoBERTa (Liu et al., 2020), LDSGM (Wu et al., 2022), XLNet-large (Kim et al., 2020), OTMT
2We report our produced results via the official code if the authors did not report results on those data.
(XLNet-large) (Jiang et al., 2022a), and Con-
trastiveIDRR (Long and Webber, 2022). Two parti-
tions of these SOTA baselines are highlighted for
comparison with our method. One partition utilizes the hierarchical information in their methods (e.g., the LDSGM and ContrastiveIDRR), and the other is to fine-tune the pre-trained language models (e.g., XLNet-large). Therefore, we include the fine-tuned T5 models to illustrate the performance gain of prompt tuning. Besides, a prompt-based method PCP (Zhou et al., 2022) and general PrefixTuning (Li and Liang, 2021), as well as Prompt Tuning (Lester et al., 2021) are included. The details of implementation are listed in A.3.

heading: 5 Experimental Result

heading: 5.1 Main Results
text: Table 2 and Table 3 summarize the main results of the PDTB 2.0 and CoNLL16 datasets, from which we derive the following conclusions. First, our method significantly outperforms all baselines and achieves state-of-the-art performance at both top and second-level classes in the IDRR task. Specifically, our method gains a considerable improvement of 6.93% second-level accuracy, 10.99% second-level F1 score, 5.96% top-level accuracy, and 9.47% top-level F1 score over the fine-tuning of the T5-large model in PDTB (Ji). It demonstrates that our method effectively utilizes the struc-
ture information and perceives the specific knowledge on the correlation of discourse relations and connectives and finally enhances the ability of T5 to undertake this challenging task. Second, the prompt-based baselines (e.g., Prefix-Tuning, Prompt-Tuning, and PCP) receive outstanding performance and perform better than the T5-large finetuning method on this task. Many works (Scao and Rush, 2021; Lester et al., 2021) have discussed the overfitting problem of T5-large fine-tuning, and this can be partially solved by prompt-tuning by updating a few learnable parameters with limited training instances. The learnable parameters of baselines and DiscoPrompt are shown in Appendix A.6. Third, the ContrastiveIDRR and our method obtain better F1 scores. This observation can support the necessity of integrating the dependencies among relations as well as connectives in the label hierarchy.
Fine-tuning a relatively pre-trained large language model (LLM) such as T5-11b requires extensive computation resources to update all trainable parameters. However, by adapting the prompt tuning-based method, the entire LLM is frozen, and only a few learnable parameters of input embeddings are required to update to obtain satisfactory performance. Therefore, we also include the performance of DiscoPrompt with the T5-11b version as a reference to explore the ability of a sizeable pre-trained language model on this IDDR task. As shown in Table 2 and Table 3, DiscoPrompt (T5-11b) easily beats other methods, achieving a 52.42% F1 score and 68.14% accuracy in the 11- class classification (second-level) task of the PDTB (Ji) and illustrating the benefits without adjusting the representations from LLMs. On the contrary,
fine-tuning T5-11b is infeasible in most single compute nodes. Considering the computation cost, we still focus on the comparison among large models.

heading: 5.2 Ablation Study
text: To better investigate the factors of DiscoPrompt, we design numerous ablations on the path prediction and the tailored hierarchical tree prompt. Table 4 reports the performance of the ablation study for our model in the PDTB (Ji).
Joint Probability for Path Prediction In our method, by estimating the likelihoods of pj1 (the top-level), pj2 (the second-level), and p j 3 (the connective) in a predicted path, the dependencies of these three masks are utilized for enhancing the ability of the pre-trained language model on this IDRR task. According to the experimental results in Table 4, we can conclude that 1) the performance of the path prediction model incorporating the signals from all three masks surpasses other models (i.e., paths forming by two arbitrary masks or one connective mask), emphasizing the significance of dependencies and effectiveness of joint prediction; 2) the predicted path model without prior knowledge of selected discriminative connectives (i.e., Path w/ Top & Second) performs the worst, which is consistent with findings in Zhou et al. (2022); 3) the predicted path model with only the connective mask (e.g., Path w/ Connective) performs consistently worse than paths adding the second mask, indicating the slight ambiguity of connectives and the necessity of the label hierarchy especially with the top. The performance gain with the complete path is at least 3.76% on average, and models associating with paths including individual connective masks can also beat the previous SOTA.
Discrete Prompt Template Two portions in our designed prompt template are in natural textual form and as discrete non-tunable tokens. The first part is the discrete tokens for the label hierarchy structure (i.e., hierarchical tree prompt), shown in Figure 3 and Figure 8. The second part is the cloze discrete prompt “The path is”. We remove the discrete tokens from the template to evaluate their importance. The performance shown in Table 4 demonstrates that the two parts of the prompt are essential for achieving satisfactory performance compared with the without manual tips (i.e., Prompt w/o Entire Prompt). When adding back the cloze discrete prompt, we do not observe the model’s ability to understand the correlations among masks for path prediction. Without explicitly injecting structural information into the hierarchical tree prompt, the performance dropped significantly, especially the second-level F1 score, dropping from 49.03% to 43.14%.
Hierarchical Tree Prompt To acquire a deeper understanding of the discrete hierarchical tree prompt, we perform experiments to gradually replace the discrete tokens with continuous ones in various elements of this hierarchy prompt. The experiments include 1) Continuous Hierarchy Prompt:
replacing the whole hierarchical tree prompt as the continuous tunable prompt with the same number of tokens, 2) Continuous Labels & Connective: only including the “-> ” and replacing other relation labels and connective as continuous tunable prompt, and 3) Continuous Connective: only replacing the textual connective to be the tunable prompt. The experimental result in Table 4 underscores the importance and effectiveness of our tailored discrete hierarchical tree prompt, which obtains at least 4.98% performance boost.
Prompt Engineering Furthermore, we conduct the prompt template searching and the parameter sensitivity on the continuous prompt length that we describe in Appendix B.2.

heading: 5.3 Label-wise F1 Scores
text: The PDTB (Ji) setting exhibits highly skewed label distributions, with only roughly 854 training instances (i.e., 6.8% of 12406 training instances) annotating as five of the 11 second-level labels. To further explore our model in four top-level relations and 11 second-level sense types on this dataset, Table 5 and Table 6 report the F1 scores (%) of the top-
level and second-level classes, respectively. In Table 5, our model outperforms all baselines in three top-level relations (i.e., Comparison, Contingency, Expansion), and most of the baselines in the Temporal relation except ContrastiveIDRR. Specifically, Table 6 illustrates that our model performs better on the Temp.Asynchronous second-level class, whereas ContrastiveIDRR is much better on the Temp.Synchrony. In Table 6, our model obtains valid predictions on most second-level classes, but all methods fail to predict Cont.Pragmatic Cause. This situation may result from the few training examples of this class being insufficient for optimal learnable parameters, and the models tend to ignore this class in the prediction process. When we check the less representative classes (i.e., Temp.Synchrony, Comp.Concession), DiscoPrompt can still make correct predictions, while PCP and ContrastiveIDRR still fail to predict neither correct ones. Moreover, we can also see the power of LLMs that the T5-11b performs remarkably better than smaller models.

heading: 5.4 Prompt Adaptation
text: For T5 Fine-Tuning To demonstrate the effectiveness of our designed template and explore whether our designed template can be used for the finetuning paradigm, we convert the data input to the tailored prompt template but with only a [MASK] for generating the entire path. The experimental results on CoNLL16 are summarised in Figure 4, and the T5-adapt boosts all metrics over vanilla T5large fine-tuning. The detailed performance and the experimental results for PDTB 2.0 are shown in Table 18 and Figure 9 in Appendix B.4.

heading: 5.5 Prompt Adaptation For ChatGPT
text: With the powerful ability of LLMs exhibited on numerous tasks, we are curious about the capabil-
ity of ChatGPT on zero-shot IDRR task. We test the ability of ChatGPT with three designed templates on the PDTB (Ji), and the performance is shown in Table 7. All designed templates obtain higher performance than the random, but still at a low region in the second level compared with supervised learning. This result reveals that IDRR is still tricky for ChatGPT and cannot solve easily at current state, consistent with the result in Chan et al. (2023). The structural path template outperforms the other two templates, proving the help of the structural form for ChatGPT to understand this task. The F1 score of each second level is shown in Figure 10 in Appendix and illustrates the effectiveness to distinguish various second-level senses among the Expansion top class. More case examples and discussions refer to Appendix B.5.

heading: 5.6 Generalization to Explicit Discourse Relation Classification Task
text: To demonstrate the generalization ability of our model, we transfer and adapt our method to the explicit discourse relation recognition (EDRR) task. We simply replace the first [MASK] between two arguments with the gold connective for each instance in EDRR. Following the previous works (Varia et al., 2019; Zhou et al., 2022), the second-level class is the same as our implicit one setting. In Table 8, our model slightly outperforms previous SOTA models on the top-level sense prediction. DiscoPrompt consistently outperforms fine-tuning under different settings, and we observe a larger margin with absenting connectives.

heading: 6 Conclusion
text: In this paper, we introduce a path prediction method for tackling the IDRR task by utilizing the hierarchical structural information and prior knowledge of connectives. Combining label structures in natural language with prompt tuning successfully takes a step further in this task as well as other generalized settings, e.g., prompt adaptation and explicit relation detection. Our model achieves new SOTA performance on PDTB 2.0 and CoNLL2016 data, and we hope our detailed discussions can help communities in discourse fields.

heading: Limitations and Future Work
text: Limited Utilized Knowledge The main limitation of our method is the limited utilized knowledge. Since our prompt tuning-based method tests on Implicit Discourse Relation Recognition (IDRR) task, the elicited knowledge only comes from the dataset of this task and the model pre-training corpora. This constraint restricts the capability owing to the reporting bias (Gordon and Durme, 2013) in the pre-training models (PLMs). Moreover, the relatively few training data of several second-level classes resulting from the highly skewed label distribution problem requires extensive knowledge to make the model understand data instances and the task. Although we impose the prior human knowledge against the IDRR task from the input template designing to the discourse connectives selection, the knowledge source still only comes from our prior knowledge and the elicited knowledge of PLMs. As a result, even our method obtains a valid score in all second-level classes except the Cont.Pragmatic Cause displayed in Table 6, some second-level senses, which are the same as previous studies, cannot receive a satisfactory performance (e.g., Comp.Concession and Expa.List). The future work for this issue is to integrate more abundant knowledge and equip the model with more vital abilities. For example, grounding the arguments pair on the relevant nodes of the knowledge graph for each data instance (Lin et al., 2019) or knowledge distillation from large language models to provides more contextual information and enhances the capability of the model on this task.
Limited Predicted Connectives Another area for improvement is the prediction of extensive connectives. Although our model includes the preselected connectives as our third layer of a designed
hierarchy tree, we do not include the ground truth of connectives as our third layer. Because including these extensive connectives to form many leaves will result in many paths (more than 100). This limitation may be addressed in future works by utilizing the pruning algorithms for reducing a lot of redundant nodes and leaves on each instance to enhance effectiveness and efficiency.

heading: Ethics Statement
text: In this work, we conformed to recognized privacy practices and rigorously followed the data usage policy. We declare that all authors of this paper acknowledge the ACM Code of Ethics and honor the code of conduct. This paper presents a method to utilize the interaction information between different layers, inherent sense label structure, and prior knowledge of connectives in the implicit discourse recognition task. The PDTB 2.0 and CoNLL-2016 dataset were used to train and assess the ability of the pre-trained language model on this task. The PDTB2.0 and CoNLL2016-Test dataset is collected from the Wall Street Journal (WSJ) articles, while the CoNLL2016-Blind dataset is derived from newswire texts, the primary language is English based and belongs to the news domain. We can foresee no immediate social consequences or ethical issues as we do not introduce social/ethical bias into the model or amplify any bias from the data. Therefore, these two datasets are not required to perform further actions to check the offensive content.

heading: Acknowledgements
text: The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520 and 16205322) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to HKMAAC and CUSBLT. We also thank the support from NVIDIA AI Technology Center (NVAITC) and the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).

heading: A Appendix for Experimental Settings

heading: A.1 DataSet
text: The Penn Discourse Treebank 2.0 (PDTB 2.0) PDTB 2.03 is a large-scale corpus containing 2,312 Wall Street Journal (WSJ) articles (Prasad et al., 2008), that employs a lexically-grounded approach to annotating discourse relations. This corpus includes three sense levels (i.e., classes, types, and sub-types) and naturally forms the sense hierarchy. In this dataset, we validate our model on two popular settings of the PDTB 2.0 dataset, which are the Ji-setting (Ji and Eisenstein, 2015) and Linsetting (Lin et al., 2009). The former one following Ji and Eisenstein (2015) to split sections 2-20, 0- 1, and 21-22 as training, validation, and test sets respectively, while the latter follows Lin et al. (2009) split sections 2-21, 22, 23 as training, validation, and test sets respectively. We evaluate our model on the four top-level implicit discourse relations and the 11 major second-level implicit discourse senses by following previous works (Wu et al., 2022; Long and Webber, 2022; Zhou et al., 2022). The data statistics of the top-level and second-level senses are displayed in Table 9 and Table 10.
The CoNLL-2016 Shared Task (CoNLL16) The CoNLL-2016 shared task4 provides more abundant annotation (e.g., second-level sense type) for shadow discourse parsing. This task includes two test sets, the PDTB section 23 (CoNLL-Test) and newswire texts (CoNLL-Blind), that comply with the PDTB annotation guidelines. Compared with PDTB 2.0, CoNLL16 includes more new class sense (e.g., Contingency.Condition) and merges several labels to annotate new labels. For example, Contingency.Pragmatic cause is merged into Contingency.Cause.Reason to remove the former type with very few samples. In this paper, we follow Wang and Lan (2016); Lan et al. (2017); Liu et al. (2020) to perform the experiments on this CoNLL-2016 dataset and validate the performance of our model in the top- and second-level sense.
3The License of the PDTB 2.0 dataset is LDC User Agreement for Non-Members, and this paper is consistent with their intended use for research purposes. This dataset download from https://catalog.ldc.upenn.edu/ LDC2008T05.
4CoNLL16 dataset download from https://www.cs. brandeis.edu/~clp/conll16st/dataset.html.

heading: A.2 DiscoPrompt Implementation Details
text: DiscoPrompt is prompt tuning upon T5-model, and we also validate our method over various model scales, including T5-base, T5-large, and T5-11b.
Figure 12 shows the heat map of highly frequent connectives on CoNLL2016, and the label words are in Table 12. Generally, the overall configuration follows the setting in Lester et al. (2021) and sets the learnable prompt length as 20. The training was implemented using cross-entropy loss with 30,000 training steps, which selects the model that yields the best performance on the validation set. We adopt an Adafactor (Shazeer and Stern, 2018) optimizer with various learning rate ranges for different dataset settings. The batch size and maximum input sequence are 4 and 350, respectively. The maximum generates sequence length of the encoder is 10. Our model is conducted on two 32GB NVIDIA V100 GPUs, except for the T5-11b scale on two 48GB NVIDIA A6000 GPUs. The running time for T5-base is around 8 hours, while T5-large is about 19 hours.
Since we are interested in the ability of our method to adopt a larger-scale model on this task, we tested the T5-11b model on various datasets. Most of the configuration is the same as the above T5-large version. The slight differences in hyperparameters are batch size is one and gradient ac-
cumulation step is 16. The running time of the T5-11b model is around 50 hours. The tailored prompt template is shown in Figure 8. The specific hyperparameters of implementation details for DiscoPrompt (T5-large) and DiscoPrompt (T5-11b) are displayed in Table 11. The frozen pre-train T5 model download from HuggingFace, and our model inheritance and modification from OpenPrompt (Ding et al., 2022).

heading: A.3 Baseline Models
text: To exhibit the effectiveness of our proposed method, we compared it with previous works on the PDTB 2.0 and CoNLL16 datasets. In this section, we mainly describe some recently published
baselines, and more baselines can be found in Table 14.

heading: Common Baselines for PDTB 2.0 and

heading: CoNLL16:
text: • RWP-CNN (Varia et al., 2019): a convolutional neural networks-based method to model word pairs in the arguments in a discourse relation.
• BMGF-RoBERTa (Liu et al., 2020): a RoBERTa-based model, which contains a robust contextualized representation module, a bilateral matching module to capture the interaction between arguments, and a global information fusion module to derive final representations for labels.
• XLNet (Kim et al., 2020): it fine-tunes XLNet model (Yang et al., 2019) for IDRR task to predict the flat label in each layer of discourse relation sense.
• T5 (Fine-Tuning) (Raffel et al., 2020): Finetune a T5-model based on specifics tailored input text in various settings with a comparison of our model. The Implementation details are described in Appendix A.5.
• Prefix-Tuning (T5) (Li and Liang, 2021): a lightweight method concatenates the tunable prefix tokens before the discrete input text, keeps language model parameters frozen, and optimizes these continuous task-specific prefix tokens. The implementation details of the PrefixTuning methods are appended in Appendix A.4.
• Prompt-Tuning (T5) (Lester et al., 2021): a vanilla Prompt Tuning-based model conditioning on a frozen model, releasing the constraints of the prompt templates from discrete to learnable prompts. The implementation details of the prompt tuning methods are appended in Appendix A.4.
• PCP (Zhou et al., 2022): a prompt-based connective prediction method for IDRR by adopting the RoBERTa model. This method utilizes the strong correlation between connectives and discourse relations to map the predicted connectives to respective implicit discourse relations.

heading: Baselines for PDTB 2.0:
text: • DER (Bai and Zhao, 2018): a model enhanced with multiple grained text representations, including character, subword, word, sentence, and sentence pair levels.
• MTL-MLoss (Nguyen et al., 2019): a multitask learning neural model that predicts labels and connectives simultaneously by leveraging the dependence between them.
• ELMo-C&E (Dai and Huang, 2019): a neural model that employs a regularization approach to utilize the external event knowledge and coreference relations.
• TransS (He et al., 2020): a TransS-driven joint learning model which translates the discourse relations in low-dimensional embedding space (i.e., TransS), and simultaneously learns the semantic features of arguments.
• CG-T5 (Jiang et al., 2021): a joint model that recognizes the relation label and generates the desired target sentence containing the meaning of relations simultaneously.
• OTMT(XLNet) (Jiang et al., 2022a): an XLNet (Yang et al., 2019) based model exploits the knowledge distillation (KD) technique for discourse relation recognition task.
• LDSGM (Wu et al., 2022): a label dependenceaware sequence generation model that integrates the global representation of an input instance, level-specific contexts, and the label dependence decoded by graph convolutional network (GCN) to obtain better label embeddings, and then employ the label sequence decoder to output the predicted labels.
• GOLF (Jiang et al., 2022b): a global and local hierarchy-aware contrastive framework, to model and capture the information from these two kinds of hierarchies with the aid of contrastive learning.
• ContrastiveIDRR (Long and Webber, 2022): a contrastive learning method for incorporating the sense hierarchy into the recognition process and using the hierarchy to select the negative examples.

heading: Baselines for CoNLL16:
text: • CoNLL Baseline (Rutherford and Xue, 2016): a neural classifier requires word vectors and a simple feed-forward training procedure.
• MTL-Attn-LSTM (Lan et al., 2017): a multitask attention-based LSTM neural network model that exploits explicit discourse relations in PDTB and unannotated external data in a multitask joint learning framework.
A.4 Implementation Details of the Prefix-Tuning and Prompt Tuning
In our paper, we implement the prefix tuning (Li and Liang, 2021) and prompt tuning (Lester et al., 2021) methods as the baselines for comparison with our model. We proposed several templates for searching for their best performance in these two methods. The experimental details for these two methods include the template and hyperparameter search. Moreover, there are 154 tokens, including textual tokens (non-tunable tokens) and tunable tokens, in our prompt template. For a fair comparison, we insert 154 tunable tokens into the respective prompt template in these two baselines.
Prefix-Tuning Following the setting of prefix tuning (Li and Liang, 2021), we implemented several designed templates on the PDTB 2.0 JI setting and the templates shown in figure 6. In these templates, we find that the prefix-prompt template three is better among all templates, and we adopted this template for further comparison with our method. The overall configuration of this model follows the settings of prefix tuning (Li and Liang, 2021). The batch size and maximum sequence length of this model are 8 and 350. The training is performed using cross-entropy loss with an Adafactor optimizer (Shazeer and Stern, 2018) and a learning rate selecting in 0.3, 0.5, 0.8 yields the best performance on the validation set, and the training steps are 30,000.
Prompt-Tuning For the prompt tuning method, we implemented several designed templates on the PDTB 2.0 JI setting and the templates shown in figure 7. In these templates, we find that the prompt
tuning template two is better among all templates, and adopted this template for further comparison with our method. The overall configuration of this model follows the settings of prefix tuning (Lester et al., 2021). The batch size and maximum sequence length of this model are 8 and 350. The training is performed using cross-entropy loss with an Adafactor optimizer (Shazeer and Stern, 2018) and a learning rate selecting in 0.3, 0.5, 0.8 yields the best performance on the validation set, and the training steps are 30,000.
A.5 Implementation Details of T5 Model Fine-Tuning
Here we provide the fine-tuning details for T5 base and large models on various datasets.
Model Input and Output In main experiments, T5-model fine-tuning as the competitive baseline, we concatenate two arguments with an “
” at the end of the sequence as input. The T5 model asked to generate the top-level labels, and the secondlevel labels with concatenating by commas (e.g., Comparison.Contrast) given the data input. For the experiments to test the transferred template on the fine-tuning paradigm, the “T5-adapt” model in section 5.4 concatenate the hierarchy tree prompt in Figure 8 before the two arguments as input. Then we concatenate a prompt message “The path is ” before the original output. Furthermore, for the setting “T5-large (fine-tune) (w/ connective)” in the EDRR task (Section 5.6), it required inserting the connectives between two arguments. Therefore, we use the text span named “FullRawText” in the dataset with an additional “
” at the end as input.
Hyperparameter Search We first conduct a preliminary experiment to determine the range of hyper-parameters. Then, we search for the learning rate within {3e− 4, 1e− 4} and warmup steps within {0, 100}. For the T5-base model, we set the training batch size as 8, and the model is evaluated with a batch size of 128 every 150 steps. For the T5-large model, the training and evaluation batch sizes are set as 16 and 64, respectively. The model is optimized with an AdamW optimizer with a linear learning rate schedule. The test performance of the model with the best validation accuracy is reported.

heading: A.6 The Approximation of Learnable Parameters
text: To show the efficiency of our method, we append the approximation of learnable parameters for all models, including our model and baselines. The approximation of learnable parameters is listed in Table 13.

heading: B Appendix for Evaluation Result and Analysis

heading: B.1 Performance of Baselines in PDTB 2.0
text: In this section, we list extensive baselines in Table 14 for comparison with our method.

heading: B.2 Ablation study on the DiscoPrompt
text: Prompt Template Searching We perform the prompt template research on our designed prompt, and all prompt searching templates are listed in Figure 8, and the performance is shown in Table 15. Our finalized optimal template inserts the connectives between two arguments to improve the textual coherence of input context and results in the PLMs easy to understand input. Therefore, this template performs better than other designed templates.
Continuous Prompt Length The continuous prompt (i.e., learnable prompt tokens) length is another factor that influences the performance of our model. Hence, we implement various prompt lengths of 10, 20, 50, and 100. The performance is in Table 16, and the optimal continuous prompt length is 20, which provides the best performance
5We use their model without a data augmentation version for a fair comparison in Table 2. This model with the data augmentation version is also appended in this table.
among all the prompt lengths and is the default prompt length for implementing other experiments. Adopting more prompt length than 20 on our method will not significantly increase this task’s performance on various evaluation metrics.

heading: B.3 Performance of Label-wise F1 Score on Top and Second level
text: The performance (F1 score%) of more baselines for comparison with our model in Top-level and Second-level shown in Table 17 and Table 19.

heading: B.4 Performance of Designed Prompt For T5 Fine-Tuning
text: The performance comparison of the T5-large finetuning with and without using our designed template on the PDTB 2.0 is displayed in Figure 9. The detailed experimental result for PDTB 2.0 and CoNLL16 dataset is shown in Table 18.

heading: B.5 Discussion and Case Example for ChatGPT
text: With the powerful ability of large language model exhibited on numerous tasks (OpenAI, 2022, 2023; Taori et al., 2023; Chiang et al., 2023; Jiang et al., 2023), we are curious about the capability of ChatGPT on zero-shot IDRR task. Hence, we test the ability of ChatGPT 6 with three designed templates on the PDTB (Ji) test set. These templates include: 1) predict the class label only, 2) predict the class label with connectives, and 3) predict the class label with connectives in a structural path form. Moreover, the input template with in-context learning highly relies on the training examples selected as the prefix instruction part of the prompt template. The performance of this model is high variance with the chosen examples vary. Therefore, this template is not taken into account in this section. The performance of the random guess model is obtained by averaging the performance of 5 runs. A prediction is regarded as wrong if ChatGPT generates the answer out of the range of label words. An interesting finding is that the ChatGPT with label-only template tends to predict many temporally related instances to the Contingency.Cause second-level sense result in poor performance on Temporal.synchrony second-level sense shown in Figure 10. The input template and two case examples are shown in Table 20 and Table 21.
6The demonstration and details of ChatGPT are on the website https://openai.com/blog/chatgpt/
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
In the limitation section after the conclusion.
3 A2. Did you discuss any potential risks of your work? In the ethics statement section after the limitation section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? In the abstract and introduction section
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The used dataset details are in section 4.1 and appendix section A.1. The utilized software is cited in appendix section A.2.
3 B1. Did you cite the creators of artifacts you used? The used datasets are cited in section 4.1 and appendix section A.1. The utilized software is cited in appendix section A.2.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The license or terms for the used datasets are stated in Appendix A.1.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The intended use for the used datasets is stated in Appendix A.1, we have not created any dataset from the existing dataset, and all dataset used is consistent with their intended use for research purposes.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In the ethics statement section after the limitation section.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The coverage of domains and the languages in the used datasets are stated in Appendix A.1.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. The relevant statistics in the used datasets are stated in Appendix A.1.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? The details of computational experiments is in section 4.2 and Appendix A.2.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? The details is in section 4.2, Appendix A.2 and A.6 .
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? The details of experimental setup is in section 4.2 and Appendix A.2 .
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The details is in section 4.2 and Appendix A.2 .
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The details of used existing packages is in Appendix A.2 .
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: The future work for this issue is to integrate more abundant knowledge and equip the model with more vital abilities. For example, grounding the arguments pair on the relevant nodes of the knowledge graph for each data instance (Lin et al. , 2019) or knowledge distillation from large language models to provides more contextual information and enhances the capability of the model on this task. Limited Predicted Connectives Another area for improvement is the prediction of extensive connectives. Although our model includes the preselected connectives as our third layer of a designed
hierarchy tree, we do not include the ground truth of connectives as our third layer. Because including these extensive connectives to form many leaves will result in many paths (more than 100). This limitation may be addressed in future works by utilizing the pruning algorithms for reducing a lot of redundant nodes and leaves on each instance to enhance effectiveness and efficiency