Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation
Year: 2023

Abstract:
Implicit discourse relation recognition (IDRR) remains a challenging task in discourse analysis due to the absence of connectives. Most existing methods utilize one-hot labels as the sole optimization target, ignoring the internal association among connectives. Besides, these approaches spend lots of effort on template construction, negatively affecting the generalization capability. To address these problems, we propose a novel Connective Prediction via Knowledge Distillation (CP-KD) approach to instruct large-scale pre-trained language models (PLMs) mining the latent correlations between connectives and discourse relations, which is meaningful for IDRR. Experimental results on the PDTB 2.0/3.0 and CoNLL 2016 datasets show that our method significantly outperforms the state-of-the-art models on coarse-grained and fine-grained discourse relations. Moreover, our approach can be transferred to explicit discourse relation recognition (EDRR) and achieve acceptable performance. Our code is released in https://github. com/cubenlp/CP_KD-for-IDRR.

heading: 1 Introduction
text: Discourse relation recognition (DRR) aims at detecting semantic relations between two arguments (sentences or clauses, they are denoted as Arg1 and Arg2, respectively). As illustrated in Figure 1, the discourse relation Contingency (denoted as sense) is held between Arg1 and Arg2, and the explicit connective so is drawn from the raw text while the implicit connective because is manually inserted by annotators. DRR is significant to many natural language processing (NLP) downstream tasks such as causal reasoning (Staliunaite et al., 2021) and question answering (Huang et al., 2021). However, compared with explicit discourse relation recognition (EDRR), implicit discourse relation recognition (IDRR) is still less accurate and practical due to the lack of connectives, which is a major challenge in current discourse analysis research.

heading: It could be a long wait. [ ] Some investors will tiptoe back in.
text: The connectives (e.g., because, so, etc.) are critical linguistic cues for identifying discourse relations. On the one hand, with the aid of explicit connectives, a simple frequency-based mapping is sufficient to achieve over 85% classification accuracy on EDRR (Xue et al., 2016). On the other hand, human annotators utilized connectives to aid relation annotation in the most popular PDTB benchmark datasets (Prasad et al., 2008; Webber et al., 2019). For instance, annotators first manually inserted a connective expression, and then determined the abstract relation in consideration of both the implicit connective and argument pairs. Therefore, several studies recognize implicit discourse relations by incorporating connective information.
Several studies incorporate connective information to recognize implicit discourse relations. One method uses the probability distribution of connectives among sense labels in the corpus (Asr and Demberg, 2020), but this requires a consistent label distribution, which is not always the case. For instance, the connective since is more likely to represent the relation contingency in the training data but temporal in the test data. Other methods predict implicit connectives before recognizing relations (Zhou et al., 2010), or project connectives and relations into the same latent space and transfer knowledge (Nguyen et al., 2019). However, these methods perform poorly because of introducing additional parameters that require training with large
5908
amounts of labeled data. Inspired by Schick and Schütze (2021), several studies exploited the advantage of prompt learning (Liu et al., 2023) to guide PLMs to predict connectives between argument pairs and then map them to corresponding discourse relations (Xiang et al., 2022; Zhou et al., 2022). However, this paradigm predicted connectives by fitting the outputs of models to one-hot hard labels, regardless of the internal association among connectives. As we all know, a discourse relation corresponds to multiple connectives, but previous studies only selected one of them as the positive sample, while other connectives with similar meanings under the same sense labels were treated as negative samples. Besides, the correlation between connectives and discourse relations utilized in these studies is a direct mapping, which is vulnerable and inaccurate. Finally, both of them spend lots of effort on template construction, which negatively affects the generalization capability.
To address above-mentioned problems, we propose a novel Connective Prediction via Knowledge Distillation (CP-KD) approach for identifying implicit discourse relations. As suggested in Hinton et al. (2015), knowledge distillation is a popular technique for training the student model to emulate the well-informed teacher model. Specifically, we first design a knowledgeable teacher model to generate meaningful soft labels that capture more associations among connectives than one-hot hard labels to guide the optimization of the student model. Secondly, we add answer hints representing the relations of arguments as input to the teacher model, which exploits the implicit knowledge between connectives and sense labels, rather than using the direct mapping relationships in the previous studies. This approach mitigates issues of connective ambiguity and the possibility of multiple similar connectives mapping to the same discourse relation. Finally, we design a simple but effective template matching the pattern of implicit discourse data, and demonstrate that simple templates can achieve acceptable performance as well. In addition, the method we propose alleviates the dependence of prompt learning on templates and has good generalization across different templates. Extensive experiments show that our proposed model outperforms prior state-of-the-art systems on the PDTB dataset by around 3%.
Our contributions are summarized as follows:
• We propose a novel Connective Prediction via
Knowledge Distillation (CP-KD) approach for the IDRR task, which achieves the SOTA performance on the PDTB 2.0/3.0 datasets and CoNLL-2016 Shared Task as well.
• Our proposed method performs label softening via knowledge distillation to capture the implicit correlations between connectives and sense labels, which previous methods ignored.
• Our method can be easily transferred from IDRR to EDRR, and experiments demonstrate that our method still performs well for EDRR.

heading: 2 Related work

heading: 2.1 Implicit Discourse Relation Recognition
text: Previous studies focused on the feature engineering of linear classifiers to classify implicit discourse relations. For example, Lin et al. (2009) was the first to consider fine-grained classification, and they further used four different feature types to characterize context and component resolution trees.
Along with the booming development of deep learning, most work designs neural networks for IDRR. For instance, Liu et al. (2021) proposed combining the context representation module and bilateral multi-perspective matching module to understand different relational semantics deeply. In addition, Wu et al. (2022) designed a label-focused encoder to learn a global representation of input instances and their level-specific context. It also uses a label-sequence decoder to output predicted labels in a top-down manner. Moreover, several methods have recognized implicit discourse relations with the aid of annotated connectives. Specifically, Kishimoto et al. (2020) proposed to introduce the auxiliary task of connectives prediction in the pre-training process and use explicit discourse relationship data for data enhancement. Kurfalı and Östling (2021) performed implicit discourse relation classification without relying on any labeled implicit relation and sidestepped the lack of data through the explicitation of implicit relations. However, these methods contradicted the original pretraining task and performed poorly on fine-grained discourse relations.
Inspired by Schick and Schütze (2021), several studies exploited the advantage of prompt learning (Liu et al., 2023) to predict connectives between argument pairs to better utilize the knowledge embedded in the PLMs. Specifically, Zhou et al. (2022) manually designed different templates that meet
the task goal and follow natural language patterns. However, this method requires a lot of effort to find a suitable template to achieve better performance. Xiang et al. (2022) developed a multi-prompt ensemble to fuse predictions from different prompting results. However, both of them predict the connective by fitting the outputs of models to hard labels (i.e., one-hot vectors), regardless of the rich semantic correlations among relations.
Another related work is Jiang et al. (2022), which uses a multi-data multi-task teacher model with explicit and implicit discourse data to optimize a single-data single-task student model. Unlike their work, which leverages knowledge distillation to transfer explicit discourse data to the student model, our work captures the intrinsic association of discourse connectives through softened category label distributions from the teacher model, thus guiding the student model.

heading: 2.2 Knowledge Distillation
text: Knowledge distillation has three prominent roles in conventional tasks: model compression, label softening, and domain migration. The principle of model compression is to transfer knowledge from one large-scale model to another lightweight model, thus enabling the model lighter without losing performance. For example, Yang et al. (2019) combined the knowledge of multiple teachers to perform question-and-answer matching. Li et al. (2020) proposed an idea to speed up Transformer model training and reasoning: training a larger model first and then compressing the model.
In knowledge distillation, the predictions from the teacher model are called soft labels, and the student model improves performance through dark knowledge, including inter-class similarity carried by the soft labels. For instance, Tang et al. (2016) found that soft labels from teacher models provide significant regularization for student models. And Cheng et al. (2020) verified mathematically that the soft label gives the student model higher learning speed and better performance than the optimization learning from the original data.
The principle of domain migration is to transfer knowledge from the teacher model to the student model in different domains. Specifically, Fang et al. (2021) found that samples from various fields shared a typical local pattern and obtained this local information for domain migration through knowledge distillation. Choi et al. (2022) extracted
domain knowledge from the existing domain pretrained models and transferred it to other PLMs through knowledge distillation.

heading: 3 Method
text: In this section, we introduce our proposed Connective Prediction via Knowledge Distillation (CPKD) method in detail. We first present the promptguided connective prediction model in Section 3.1 and then describe the overall framework of our CP-KD approach in Section 3.2.

heading: 3.1 Prompt-Guided Connective Prediction
text: The prompt-guided connective prediction method aims to predict the most probable connective between arguments and then map it to the corresponding sense label. As illustrated in Figure 2, it has three main processes, including template construction, connective prediction, and answer mapping.
Template Construction: In this module, we give different inputs for the teacher and student models. For the student model, given a pair of arguments, we transfer them to xprompt−s with the template:
xprompt−s = T(xArg1,xArg2), (1)
where xArg1 and xArg2 correspond to two arguments, respectively (as shown in Figure 1) and T represents template function. In the PDTB corpus, almost all implicit discourse data satisfy the "Arg1 connective Arg2" sequence order, where the connective is manually inserted by annotators. Therefore, we design a simple but effective template "Arg1 [MASK] Arg2" for our main experiment, where the symbol [MASK] represents the masked token in place of the predictable connective.
For the teacher model, we add the answer hint as input and combine it with the given argument pairs to xprompt−t with a new template:
xprompt−t = T(xArg1,xArg2,xhint), (2)
where xhint represents the specific sense label, such as Contingency. For simplicity and clarity, we use "Arg1 [MASK] Arg2 Answer: sense" as the teacher template.
Connective Prediction: Then we feed xprompt to the RoBERTa (Liu et al., 2019) model to obtain the representation of [MASK] token hmask, and input the token into MLMHead model to acquire scores emask of each word in its vocabulary V .
emask = MLMHead(hmask). (3)
According to the hierarchy sense labels and implicit connectives, we manually select a discrete answer space V a, which is a subset of PLM’s vocabulary V . During the training, a softmax layer is applied on emask to normalize it into probabilities:
Pi = exp (ei)∑|V a|
k=1 exp (ek) , vi ∈ V a, (4)
where |V a| is the size of vocabulary V a. Afterwards we use cross-entropy to calculate the loss between the model prediction and the selected golden connective:
LMLM(x) = − 1 |M| ∑
i∈M yi logPi, (5)
where M denotes the set of masked tokens and yi represents the golden label.
Answer Mapping: Finally, we map the predicted connective (e.g., because) to the corresponding sense label (e.g., Cause). For implicit discourse relation data, each sample has been annotated with the connective appropriate to it on PDTB and CoNLL16 datasets (detailed in Section 4.1). However, the number of connectives marked in the original samples is large, and the ambiguity is high. As a result, we select the most frequent and less ambiguous connectives as the answer words. At the same time, we only select those tokenized connectives with a single token as answer words since most masked PLMs predict a single word. We present the final answer sets we select on the PDTB 2.0/3.0 and CoNLL16 datasets in Appendix B.

heading: 3.2 Overall Framework of CP-KD
text: As illustrated in Figure 3, our proposed CP-KD approach consists of two branches: a teacher model T , which aims to combine soft type constraints between connectives and sense labels with promptguided connective prediction model to instruct the optimization of the student model, and a student(distilled) model S, which is forced to produce
vectorized outputs that are similar to the results of the teacher model.
In the training stage, the optimization goal of the teacher model is to correctly predict the golden connective when adding sense words as answer hints. To enable the teacher model to predict connectives without relying on "reciting answers," we select a fraction of the random samples to add sense words as hints (detailed in Section 4.6.1).
Meanwhile, the student model requires to serve testing scenarios where extra sense labels are missing. Therefore, the student is expected to tap the deep semantic relationships of argument pairs with the guidance of a knowledgeable teacher. As shown in Figure 3(a), in the training stage, the student model S is required to match not only the groundtruth one-hot labels but also the probability outputs of the teacher model T :
Ls = αLSGT + (1− α)τ2LKD, (6)
where α is the coefficient to trade off such two terms and τ is the temperature rate parameter used to alleviate category imbalance. In addition, LSGT is the ground-truth loss using one-hot labels to predict connectives, and LKD is the knowledge distillation loss utilizing the Kullback-Leibler divergence(Hershey and Olsen, 2007) to quantify the difference of output distribution from student’s soft predictions to teacher’s soft labels:
LSGT = − 1 |K| K∑
i=1
yi log exp (ei)∑|Va|
k=1 exp (ek) , (7)
LKD = K∑
i=1
P̃T (i) log ( P̃T (i)/P̃S(i) ) , (8)
where yi is the golden label, K is the size of instance, P̃ = softmax ( Z̃/τ ) , and Z̃ is the pre-
softmax logits output by the model. As shown in Figure 3(b), in the inference stage, the well-trained student model aims to predict connectives between a pair of arguments and then map it to corresponding discourse relations.
It is worth mentioning that the inclusion of temperature rate τ in the softmax layer contributes to flattening the distribution, narrowing the gap between two models and making the distillation focus on whole logits, as illustrated in Figure 3(c). Furthermore, as seen in Figure 3(d), soft labels output by the teacher model carry more information
among connectives than one-hot labels. For example, the connective so is semantically similar to thus, yet hard labels do not carry such information.

heading: 4 Experiment

heading: 4.1 Dataset
text: The Penn Discourse Treebank (PDTB 2.0/3.0) PDTB corpora are annotated with information related to discourse semantic relation. Among them, PDTB 2.0 (Prasad et al., 2008) contains 2312 Wall Street Journal (WSJ) articles, while PDTB 3.0 (Webber et al., 2019) has made a series of modifications based on Version 2, including annotation of 13,000 additional tokens and incorporation of new senses. We follow (Ji and Eisenstein, 2015) to take the sections 2-20 as the training set, 0-1 as the development set, and 21-22 as the testing set. We evaluate our model on both coarse-grained and fine-grained discourse relations. Table 1 shows the statistics of the top-level senses. We introduce the CoNLL16 dataset in Appendix A.

heading: 4.2 Baselines
text: To validate the effectiveness of our method, we compare our approach with the advanced models in recent years. First of all, we select some strong baselines based on the neural network, including ESDP (Wang and Lan, 2016), MANN (Lan et al., 2017), and RWP-CNN (Varia et al., 2019). Their work mainly focused on the top-level senses of PDTB 2.0 and CoNLL16 cross-level senses. Secondly, we compare our method with competitive baselines based on PLMs, such as HierMTN-CRF (Wu et al., 2020), BERT-FT (Kishimoto et al., 2020), BMGF-RoBERTa (Liu et al., 2021) and LDSGM (Wu et al., 2022). These methods achieve impressive performance at the fine-grained secondlevel senses with the help of large-scale PLMs. Finally, we compare our approach with the latest
work PCP (Zhou et al., 2022) and ConnPrompt (Xiang et al., 2022). Both utilize the strategy of prompt learning to predict connectives and achieve stateof-the-art performance on PDTB 2.0 and PDTB 3.0 datasets, respectively. Since almost all previous methods were not experimented on PDTB 2.0/3.0 and CoNLL16 datasets at the same time, to comprehensively evaluate the performance, we choose several competitive models in the last three years (including BMGF-RoBERTa, LDSGM, PCP, and ConnPrompt) to re-implement on three datasets.1

heading: 4.3 Implementation Details
text: In this work, we use RobertaForMaskedLM 2 as the backbone of our method, where RobertaEncoder is to obtain context representation of inputs and RobertaLMHead is to acquire each vocabulary token prediction score for [MASK] token position.
We adopt AdamW optimizer (Loshchilov and Hutter, 2017) with the learning rate of 1e−5 to update the model parameters and set batch size as 16 and accumulated gradients as 2 for training and validation. Since the knowledge distillation method is sensitive to hyperparameters, we use the optimization algorithm of grid search to explore the practical effect under different parameters, where α takes value from 0.3 to 0.7 and τ from 1 to 5. All our experiments are performed on one RTX 3090.
1Since the LDSGM model utilizes the hierarchical relationship between top-label and second-level, it does not apply to the cross-level recognition of CoNLL16. We mark the data we re-implement with a superscript.
2https://huggingface.co/docs/ transformers/model_doc/roberta
All other parameters are initialized with the default values in PyTorch Lightning3, and our model is all implemented by Transformers4.

heading: 4.4 Experimental Results and Analysis
text: We first evaluate our model on the coarse-grained top-level and fine-grained second-level senses of PDTB 2.0/3.0 (denoted as PDTB2-Top, PDTB2Second, PDTB3-Top, and PDTB3-Second, respectively) with Macro F1 score and accuracy value. Then we conduct cross-level classification on the CoNLL16 dataset and consider accuracy as the primary metric, denoted as CoNLL and Blind for the test and blind-test set.
Table 2 shows the main results, from which we can reach the following conclusions. First, our method achieves the new SOTA performance with substantial improvements on almost all implicit discourse recognitions, which proves the superiority and generalization of our approach. Specifically, when considering accuracy, it obtains 3.82%, 2.98%, 1.81%, 2.64% and 2.85% improvements over the best results of previous baselines (Part 3) on PDTB2-Top, PDTB2-Second, CoNLL, PDTB3Top, and PDTB3-second classifications, respectively. In terms of F1, it also performs consistently better than previous models. Second, compared with the latest work PCP and ConnPrompt, the most significant improvement of our approach is utilizing knowledge distillation to obtain implicit
3https://github.com/Lightning-AI/ lightning
4https://github.com/huggingface/ transformers
relationships between connectives and sense labels instead of using direct mapping relationships. The experimental results prove the meaningful soft labels generated by the teacher model contribute to recognizing the implicit relations between argument pairs. Third, it can be observed that our CP-KDbase approach outperforms the PCPlarge method on almost all datasets, which proves that knowledge distillation supports the student models to obtain significant performance gains, even over larger models. (See Appendix C for more analysis.)

heading: 4.5 Ablation Study
text: To evaluate the effects of different components, we compare CP-KD with its variants: 1) w/o KD. In this variant, we remove the teacher model and only remain the student model for connective prediction; 2) w/o MLM. In this variant, the teacher model predicts connectives through [CLS] of the PLMs. 3) w/o hint. In this variant, we remove answer hints of the teacher model. We intend to explore whether adding answer hints for the teacher model contributes to learning the deep correlations between connectives and sense labels and thus help implicit discourse relation recognition.
From Part 1 of Table 3, we can observe that our CP-KD model consistently exhibits better performance than their corresponding variants across both coarse-grained and fine-grained labels. Specifically, the knowledge distillation module brings the most significant performance improvements, with about 5% gains in F1 and Acc. metrics on almost all datasets. Moreover, the performance decreases by about 2% when CP-KD w/o MLM as the reference, which proves the prompt-guided method outperforms the conventional pre-train and finetuning paradigm model. Finally, the performance improvement on fine-grained classification is more significant than coarse-grained when the teacher model adds answer hints, which demonstrates that answer hints can guide the teacher model to explore the implicit relationships between connectives and
sense labels accurately.

heading: 4.6 Hyperparameter Tuning

heading: 4.6.1 Proportion of Answer Hints
text: In this section, we explore the appropriate ratio for introducing answer hints. As shown in Table 4, the optimal balance of selected answer hints is 10% for the PDTB 2.0/3.0 datasets. When the ratio is lower, it is difficult for the teacher model to discover the relationship between connectives and sense labels. The teacher model is more inclined to recite the answers when the proportion is higher. We can imagine the teacher model as an experienced professor who teaches the best students when it has seen some samples instead of remembering all the answers. Moreover, the optimal proportion for the CoNLL16 datasets is 40%, which indicates that fine-grained classification requires more cues than coarse-grained to uncover the implicit relationship between connectives and sense labels.

heading: 4.6.2 Influence of Hyperparameter in KD
text: As we all know, the knowledge distillation algorithm is sensitive to hyperparameters and random seeds. To explore the effect of hyperparameters, we experiment with ten consecutive random seeds varying α from 0.3 to 0.7 and τ from 1 to 5 on the PDTB 2.0 top-level senses.
As we can observe from Figure 4, the average performance is significantly better when α is smaller, demonstrating that the teacher model’s soft labels can carry more information than one-hot
hard labels. In addition, the average Acc. and F1 scores reach their highest values when both α and τ are small, which proves that when the student model prefers the knowledge of the teacher model, τ needs to be tuned down to prevent the effect of negative labels.5

heading: 4.7 Case Study
text: Figure 5 showcases the confusion matrices of both the ConnPrompt (Xiang et al., 2022) and the CPKD models, tested on the PDTB 2.0 second-level senses. The matrices highlight ConnPrompt’s challenge in differentiating between closely related categories, namely Comp.Contrast and Exp.Conjunction, as well as Cont.Cause and Exp.Restatement. This confusion emphasizes the criticality of profound semantic comprehension for precise implicit discourse relation recognition.
Contrarily, CP-KD, leveraging the benefits of knowledge distillation, displays superior capabilities in discerning these nuanced differences. This demonstrates that a simplistic reliance on surfacelevel lexical or syntactic features is inadequate, and a deeper understanding of semantics is necessary. We present this through the following examples:
• Example 1: ConnPrompt confuses Cont.Cause with Exp.Restatement.
Arg1: He was right.
Arg2: By midday, the London market was in full retreat.
• Example 2: ConnPrompt erroneously identifies Comp.Contrast as Exp.Conjunction.
Arg1: Amcore, also a bank holding company, has assets of $1.06 billion.
Arg2: Central’s assets are $240 million.
5See the appendix D for details about the results.
The examples provided above underscore CPKD’s enhanced capability to comprehend the semantic relationships between pairs of arguments. This enhancement can largely be attributed to the integration of knowledge distillation within CP-KD, which fosters a deeper understanding of discourse relations and connectives. Despite the model’s praiseworthy performance, we recognize the potential for further optimization and exploration. Specifically, the model requires improvement in handling few-shot categories such as Exp.List and Temp.Synchrony. To bolster the model’s overall predictive precision and robustness, we propose increasing its competency in managing underrepresented senses. This can be achieved by enriching the training set with additional instances of these categories, enhancing the model’s familiarity with these senses, thereby augmenting its predictive capabilities.

heading: 5 Discussion

heading: 5.1 Generalization to Other Prompt Template
text: Previous studies proved that templates have different impacts on the prediction results of connectives
(Xiang et al., 2022; Zhou et al., 2022). Therefore, in this section, we are tempted to verify the generalization of our method on different templates. Specifically, Zhou et al. (2022) found a relatively best template for connective prediction after abundant experiments. Xiang et al. (2022) designed three prompt templates and made a decision fusion of majority voting as multi-prompt ensembling for final relation sense prediction. For a fair comparison, we replace the templates of our approach to verify the effectiveness of knowledge distillation.
As shown in Table 5 and 6, our method has successfully generalized different templates. It is worth mentioning that the general template used in this paper is precisely the same as the first template in ConnPrompt (Xiang et al., 2022). When we use multi-template fusion like it, our method achieves better performance on the PDTB 3.0 dataset.

heading: 5.2 Generalization to Explicit Discourse Relation Recognition
text: Inspired by the attempt of section 5.1, we transfer our method to the EDRR task. Similarly, we design a simple template in line for the explicit discourse relation recognition via knowledge distillation (KD-EDRR). The new template is as follows:
• <start> Connective <end> Arg1 [MASK] Arg2
where the Connective represent connectives that appear in the original text but not in Arg1 or Arg2. In addition, <start> and <end> are marker tokens used to guide the position of connective. Meanwhile, we use the [CLS] token of the masked language model to predict the sense directly, and we introduce the symbol [MASK] to predict connective, which is regarded as an auxiliary task for mining the implicit relationships between connectives and sense labels.
As shown in Table 7, the variant of our method KD-EDRR achieves the new state-of-the-art performance on the top-level senses of PDTB 2.0 for the EDRR task, which effectively demonstrates the generalizability of our approach.

heading: 6 Conclusion
text: In this paper, we propose a novel connective prediction via knowledge distillation approach for coarsegrained and fine-grained implicit discourse relation recognition. Experimental results demonstrate that our method achieves state-of-the-art performance on the PDTB 2.0/3.0 datasets and the CoNLL-2016 Shared Task. Furthermore, our proposed method fully uses the correlation between connectives and sense labels and achieves good generalization on different templates. Finally, we experimentally prove that our approach can be transferred from IDRR to EDRR and still performs well for EDRR. We will later explore the applicability of our approach to some Chinese discourse relations datasets for coarse-grained and fine-grained DRR.
Limitations
In this section, we will point out the limitations of our work, which can be summarized in the following two aspects.
Firstly, in the step of answer mapping (Section 3.1), we only select those connectives that are tokenized with a single token as answer words, since most masked PLMs predict only a single word. Therefore, those connectives tokenized with multiple tokens will be replaced by the most frequent answer word with the same subtype-level sense tags. We believe that this approach will filter out several meaningful connectives as answer words. In the future, we will utilize the generative
model to predict the connectives between argument pairs, which can decode multiple tokens at a single mask position.
Secondly, in section 5.1, we can observe that multi-prompt ensembling is effective for fusing multiple single-prompts for implicit discourse relation recognition. In the future, we will explore multi-teacher knowledge distillation method for the IDRR task, here teacher models are trained with different templates. In this way, we can take advantage of the different prompt templates.

heading: Acknowledgement
text: We appreciate the support from National Natural Science Foundation of China with the Main Research Project on Machine Behavior and HumanMachine Collaborated Decision Making Methodology (72192820 & 72192824), Pudong New Area Science & Technology Development Fund (PKX2021-R05), Science and Technology Commission of Shanghai Municipality (22DZ2229004), Shanghai Trusted Industry Internet Software Collaborative Innovation Center and East China Normal University International Conference Grant Programme.

heading: B Answer Sets on Three Datasets
text: In this section, we present the answer sets we select on PDTB 2.0/3.0 and CoNLL16 datasets, as illustrated in table 8, 9 and 10. In addition, we found that there are several data samples with two senses. In our data statistics and experiments process, we uniformly considered the first sense of these samples as their golden label for avoiding ambiguity.

heading: C Performance on Fine-grained IDRR
text: To better evaluate the performance of our method on fine-grained implicit discourse relation recognition, we compare it with three previous competitive models at each second-level sense of PDTB datasets 6. As exhibited in Table 11 and 12, our
6Given the test set of PDTB 2.0 only covers 11 types of discourse relations, we restrict our results to a statistical analysis of these 11 discourse relations in this study. The same procedure was followed for the PDTB 3.0 dataset.
CP-KD method supersedes the prior state-of-theart models in the majority of second-level senses, barring a few exceptions such as Exp.List and Exp.Conjunction.
Notably, our approach procures significant enhancements in several categories already demonstrating robust performance, such as Comp.Contrast and Exp.Instantiation. The improvements in these categories indicate that the novel approach of transforming the implicit discourse relation recognition task into a connective prediction task, followed by employing knowledge distillation to capture intrinsic connective associations, is highly effective.
Furthermore, the CP-KD method demonstrates an exceptional capacity to handle complex implicit relations, as evidenced by its superior performance in categories like Comp.Similarity and Cont.Condition in the PDTB 3.0 dataset, and Comp.Concession in the PDTB 2.0 dataset. This underlines the effectiveness of a combined approach of Prompt Learning and knowledge distillation in tackling intricate implicit discourse relations.
Additionally, our CP-KD method maintains a high degree of stability across various discourse relations, as shown by its consistently competitive performance across different relation types. This attribute reaffirms the CP-KD method’s robust recognition capability across a diverse range of implicit discourse relations.

heading: D Results of Different Hyperparameters
text: In deep learning models, particularly those employing techniques like knowledge distillation, perfor-
mance can be sensitive to the choice of hyperparameters and random seed (Cho and Hariharan, 2019). To scrutinize this effect, we conducted experiments with ten consecutive random seeds, varying the hyperparameters α in the range of 0.3 to 0.7 and τ from 1 to 5 on the PDTB 2.0 top-level senses. Table 13 and table 14 show the average and overall results, respectively, for different combinations of hyperparameters and random seeds.
Variations in the results can be attributed to the stochastic nature of deep learning model training and the specific dynamics induced by knowledge distillation. The balance between learning from soft targets (teacher’s predictions) and hard targets (original ground truth labels) - governed by the hyperparameters - and the model weights initialization (controlled by the random seed) can significantly influence the optimization trajectory and final model performance.
While initial results were reported with a single random seed, we believed it necessary to demonstrate the effect of these variables on our CP-KD method. Despite the observed fluctuations, our model outperforms the state-of-the-art on average, attesting to the robustness and superiority of our approach. This analysis underscores the importance of thorough hyperparameter studies in future research for ensuring reproducibility and robustness of results in implicit discourse relation recognition. Other than the results in this section, experiments were performed with the first random seed.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
In Section 6 Conclusion and Limitations.
7 A2. Did you discuss any potential risks of your work? Our paper is a foundational research. In our paper, we aims to utilize knowledge distillation to mine the internal correlations between connective and sense labels to address the implicit discourse relation recognition. We cannot think of any potential risks of our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1 Introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? In Section 4 Experiment.
3 B1. Did you cite the creators of artifacts you used? In Section 4.3 Implementation Details.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In Section 4.2 Baselines.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Section 3 Methods. Apache License 2.0 gives permission on Commercial use, Modification, Distribution, Patent use and Private use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The dataset we applied is a commonly used open-source benchmarks datasets in the field of shallow discourse parsing.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In Section 3 Methods and Section 4.1 Datasets.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Section 4.1 Datasets.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? In Section 4.4 Experimental Results and Analysis.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In Section 4.3 Implementation Details.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? In Section 4.3 Implementation Details.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section 4.4 Experimental Results and Analysis.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Section 4.5 Ablation Study.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This analysis underscores the importance of thorough hyperparameter studies in future research for ensuring reproducibility and robustness of results in implicit discourse relation recognition. Other than the results in this section, experiments were performed with the first random seed. ACL 2023 Responsible NLP Checklist